🎬 AI Omnichannel Content Automator (Proprietary Engine)

Status: 🔒 Proprietary / Internal Tooling. Source code is closed.

This repository documents the architecture, logic, and engineering systems behind a fully autonomous, end-to-end Short-Form Content (SFC) generator. The engine orchestrates Large Language Models (LLMs), Image Diffusion, Premium TTS, and a highly complex programmatic FFmpeg rendering pipeline to autonomously write, edit, and publish high-retention video content to YouTube and Instagram.

⚙️ Core Engineering & System Architecture

1. AI Content & Prompt Engineering Engine

The script utilizes Cloudflare Workers AI (Qwen 3.8-27b) for scriptwriting and metadata generation.

Strict Output Formatting: To prevent JSON parsing errors or hallucinated conversational text, the LLM is restricted via a heavily engineered prompt. It enforces output entirely in custom XML-style tags (e.g., [SCRIPT], [TITLES], [TAGS], [IMAGE]).
Anti-Reasoning Injection: The system dynamically injects instructions to suppress Chain-of-Thought (CoT) internal monologues, forcing immediate, deterministic outputs.
Bilingual Constraint: The engine forces the LLM to write the core script in native Hindi (Devanagari) to ensure the ElevenLabs TTS reads it with accurate native inflection, while simultaneously generating SEO metadata and Image Prompts in English.
Visual Generation: Extracts the English [IMAGE] prompt and feeds it into FLUX-2 via Cloudflare, generating 9:16 hyper-realistic base assets.

2. The FFmpeg Mathematical Rendering Engine

The crown jewel of this system is its dynamic video compilation logic. Instead of static templates, the engine mathematically calculates a unique FFmpeg filter_complex string at runtime based on the TTS audio length.

Audio RMS Beat-Detection & Dynamic Zoom: The engine extracts a temporary WAV file, reads the raw audio frames in 100-sample windows, and calculates the Root Mean Square (RMS). It isolates timestamps where the RMS peaks above a dynamic threshold (breaths, hard consonants, or musical beats). These timestamps are injected into a dynamic FFmpeg zoompan expression (e.g., if(between(time, x, y), zoom+0.005, 1.00)), creating subtle, algorithmically perfectly-timed visual punch-ins.
Advanced Timeline Math & Duration Matching: The engine calculates the required visual duration: Target_Duration = Audio_Delay + TTS_Duration + Post_Audio_Delay.
- Shortage Handling: If the provided visual media is shorter than the target, the engine automatically calculates a setpts multiplier (e.g., setpts=1.45*PTS) to seamlessly slow-mo the footage to match the audio perfectly.
- Excess Handling: If footage is too long, the engine securely trims or calculates random looping patterns to ensure the final output strictly adheres to the under-60-seconds Shorts algorithm limit.
Chroma Key (Green Screen) Compositing: Reads from a master VFX JSON pool. It applies colorkey filtering with adjustable similarity and blend parameters, processes the alpha channel via colorchannelmixer for custom transparency, and precisely overlays the VFX at specific configured timestamps.

3. Asynchronous Omnichannel API Publishing

The publishing module is a highly resilient, state-aware engine designed to handle massive file uploads and strict API constraints.

The Dropbox Ephemeral CDN System: Meta’s Graph API strictly requires media to be hosted on a public URL; it does not accept local file streams. To bypass server hosting requirements, the system uses the Dropbox API. For files larger than 4MB, it utilizes UploadSessionCursor to perform secure, chunked uploads, preventing RAM exhaustion. It then generates a direct download link and feeds this to Instagram.
Meta/Instagram Graph API (v26.0) Polling System:
1. Creates a media container using the Dropbox URL.
2. Enters a while loop, continuously polling Meta's servers for the status_code.
3. Handles edge cases (e.g., Meta throwing generic errors for Images which don't return status codes, or EXPIRED container tokens).
4. Upon a FINISHED status, executes the final media_publish command. Includes logic for Carousel (multi-child containers) and single Reels.
5. Auto Token Rotation: Checks if the Instagram long-lived access token is older than 25 days and automatically hits the Meta refresh endpoint to prevent expiration.
YouTube Data API v3: Handles OAuth2 token refresh, applies metadata, and intelligently routes the video (If duration is under 60s, appends #Shorts; otherwise, uploads as a standard YouTube video). Utilizes MediaFileUpload with resumable=True.

🔄 Execution Workflow

graph TD;
    A[CLI: Project Init] --> B[LLM Prompted for Script & Metadata];
    B --> C[XML Extractor parses Output];
    C --> D[ElevenLabs Premium TTS Gen];
    D --> E[Calculate Audio RMS Windows for Beats];
    E --> F[Calculate Required Visual Duration];

    subgraph Dynamic FFmpeg filter_complex Builder
    F --> G[Apply Media Stretch / Loop logic]
    G --> H[Inject Zoompan equations based on RMS]
    H --> I[Apply Chroma-Key VFX overlays]
    I --> J[Apply Transitions, BGM Ducking & EQ]
    end

    J --> K[FFmpeg Renders Final MP4];
    K --> L{Human QA Verification};

    L -- Approved --> M[Dropbox API Chunked Upload];
    M -- Direct Link Generated --> N[Meta API Container Init];
    N --> O{Poll Meta Servers};
    O -- FINISHED --> P[Publish to Instagram];
    L -- Approved --> Q[Resumable Upload to YouTube];

🛠️ Resilience, State Management & Mobile Optimization

State-Saving Flags (Fault Tolerance): Video rendering and API uploads can fail due to network drops. Every project utilizes a localized config.json containing skip flags (skip_content_generation, skip_audio, skip_video). If the script crashes during the YouTube upload, restarting the script will bypass AI generation and Video rendering, immediately resuming the upload process.
Termux / Android Wake-Lock Integration: To allow this pipeline to run natively on Android devices via Termux, the script utilizes a custom TermuxWakeLock context manager. It triggers termux-wake-lock via the Termux API during heavy FFmpeg CPU loads or long network uploads to ensure the Android OS battery optimizer does not kill the process in the background, releasing the lock (termux-wake-unlock) when finished.
Safe JSON Handling: The system utilizes custom safe_load_json and strict_config type-casting wrappers. This ensures that if a user inputs a string instead of a float in the configuration files, the script halts securely with a detailed error log rather than failing silently halfway through a complex FFmpeg render.

📂 Configuration Architecture

The engine utilizes a hierarchy of configuration files to maintain both global consistency and per-project granularity.

1. Global Master Pools (`/config/video/`)

Contains master configurations that apply globally:

effects.json: Stores pools of FFmpeg color-grading filters (eq=contrast=1.05:saturation=1.1). The script randomly schedules "short-time bursts" of these filters mathematically to retain viewer attention.
transitions.json: Master list of FFmpeg xfade transitions (wipeleft, circlecrop, dissolve).
vfx.json: Master definitions of green-screen assets, containing exact hex color codes for chroma-keying, similarity thresholds, and blend settings.

2. Project-Specific Overrides (`projects/{name}/config.json`)

Every generated video isolates its assets and logic into its own folder. The local configuration allows granular control:

{
    "media_type": "m",
    "duration_mode": "audio",
    "shortage_fill": "slow", 
    "mixed_media": [
        "intro_hook.mp4",
        "ai_generated_image.png, 4.5", 
        "b_roll.mp4"
    ],
    "voice_effect_name": "Podcast Presence",
    "bgm_effect_name": "Cinematic Muffled",
    "vfx_layers": [
        {
            "vfx_name": "Glowing Star Overlay",
            "start_time": 2.5,
            "end_time": 10.0,
            "mode": "slow",
            "transparency": 0.85
        }
    ]
}

Notice how audio EQ profiles ("Podcast Presence"), VFX scheduling, and mixed-media custom durations are fully modular per video.

📊 Summary of Tech Stack

Component	Technology / Concept
Language	Python 3.8+ (Strictly typed config parsers, OOP contexts)
Video Engine	FFmpeg (Advanced `filter_complex`, `xfade`, `zoompan`, `colorkey`)
Audio Processing	Waveform analysis (RMS calculation via `wave` and `struct`)
AI LLM	Cloudflare Workers AI (`Qwen 3.8-27b`) via REST API
AI Vision	Black Forest Labs `FLUX-2` via Cloudflare Workers AI
TTS Engine	ElevenLabs API (w/ live token tracking & stability tuning)
Cloud CDN	Dropbox SDK (Chunked Session uploads)
Social APIs	Google/YouTube Data API v3 (OAuth2), Meta Graph API v26.0
DevOps	Termux API integration (Background Wake Locks)

Developed as a demonstration of advanced API orchestration, AI prompt engineering, and programmatic video rendering.

Automated Social Media Post