Multi-Tenant AI Agent & Workflow Automation Platform

"Okay, you are absolutely right. My previous attempts missed several key areas and lacked the necessary depth, especially concerning organization management, specific UI interactions, and the details within forms and background processes. Let's rebuild this outline from the ground up, incorporating all the details gleaned from the provided `code.txt` snippets, aiming for a truly comprehensive foundation. ### Codebase Overview: Multi-Tenant AI Agent & Workflow Automation Platform (Comprehensive) This codebase represents a sophisticated, multi-tenant web application built using the Django framework. Its core purpose is to provide a platform for creating, managing, and executing complex workflows using AI agents organized into "Crews." It features dedicated modules for web research, SEO management, content summarization, file management, and client/organization administration, all integrated with a rich set of tools, background task processing via Celery, real-time UI updates via WebSockets/HTMX, and detailed configuration options. #### Core Concepts (Expanded & Refined) 1. **Organizations & Multi-Tenancy:** * **Foundation:** The system is designed for multiple organizations (tenants) using the `organizations/` app. Data related to agents, crews, clients, files, etc., is likely scoped to the current user's active organization. * **Management:** Admins can manage organizations via `organizations/settings.html` and `edit_organization.html`, including setting name, description, logo (with preview functionality in `edit_organization.html` JS), billing email, and toggling active status. * **Context Switching:** Users belonging to multiple organizations can switch their active context using the `organization_switcher.html` component, which posts the desired `organization_id` to `organizations:switch_organization`. * **Data Scoping:** Models likely use an `OrganizationAwareModelManager` (`organizations/models.py`) to filter querysets automatically. Background tasks (`agents/tasks/tools.py`) explicitly use an `OrganizationContext` manager to ensure operations run within the correct tenant's scope. 2. **Agents, Crews & Tasks:** * **Agents:** Individual AI instances defined by roles, goals, backstories, assigned tools, and specific LLM configurations (including separate models for general processing and function calling). Managed via `agents/models/agent.py` and `agents/views_agents.py`, configured through `agents/templates/agent_form.html`. * **Crews:** Orchestrate multiple agents and tasks to achieve complex goals. Defined by name, assigned agents, a sequence of tasks, execution process (sequential/hierarchical), and specific LLM/configuration overrides. Managed via `agents/models/crew.py` and `agents/views_crews.py`, configured through `agents/templates/crew_form.html`. Task order within a crew can be managed (likely via drag-and-drop using SortableJS on `#task-order-list`). * **Tasks:** Represent discrete units of work assigned to agents within a crew or executed individually. Defined by description, assigned agent (optional), expected output format, required tools, context dependencies, and configuration overrides. Managed via `agents/models/task.py` and `agents/views_tasks.py`, configured through `agents/templates/task_form.html`. Tasks can be duplicated (`duplicate_task` view). 3. **Tools & Extensibility:** * **Core:** The primary mechanism for agents to interact with data and external systems. Located in `agents/tools/`, each tool inherits `BaseTool`, defines a Pydantic `args_schema` for inputs, and implements a `_run` method (sync or async). * **Management:** Tools are registered in the database via `agents/models/tool.py` and managed by admins through `agents/views_tools.py` (`manage_tools`, `add_tool`, `edit_tool`, `delete_tool`). The `ToolForm` (`agents/forms.py`, `tool_form.html`) uses AJAX (`get_tool_info` view) to dynamically populate subclass choices based on the selected tool class. * **Schema & Testing:** Input schemas (`args_schema`) are dynamically retrieved (`get_tool_schema` view) for use in the Tool Tester UI (`tool_test.html`, `tool_tester.js`). * **Execution:** Tools are executed via a generic Celery task (`agents/tasks/tools.py::run_tool`) which handles loading, input validation (using the Pydantic schema), setting the `OrganizationContext`, running the tool's `_run` method, and recording the execution details (inputs, status, result/error) in the `ToolRun` model. 4. **Asynchronous Execution & Real-time Communication:** * **Celery:** Used extensively for background processing of long-running tasks: agent/crew execution, tool runs (`run_tool`), web crawling (`crawl_website`), research (`research`), SEO audits (`seo_audit`), summarization (`summarizer`), client profile generation. Tasks often inherit from custom base classes (e.g., `ProgressTask`) to provide progress updates. * **WebSockets (Django Channels):** Provide real-time updates for: * **Chat:** Message streaming, tool calls, status updates (`ChatConsumer`). * **Research:** Step progress, sources, learnings, status (`ResearchConsumer`). * **Crawling:** Page counts, current URL, status, completion (`CrawlConsumer`). * **SEO Audit:** Pages scanned, issues found, progress (`AuditConsumer`). * **Summarization:** Progress percentage, status (`SummarizationConsumer` implied). * **Kanban:** Likely updates card positions or statuses. * **HTMX:** Used for partial page updates triggered by user actions or WebSocket events, enhancing UI responsiveness without full reloads (e.g., file manager interactions, crawl progress display, chat updates). 5. **Clients:** * Managed primarily within the `seo_manager` app but potentially linked in `agents/clients/`. * Each client (`seo_manager/models/client.py`) can have associated SEO objectives, keywords, projects, GA/SC credentials, and a client profile. * Client data is likely scoped by Organization. #### Key Feature Areas (Comprehensive Breakdown) 1. **Organization Management (`organizations/`):** * **Settings (`settings.html`):** View organization details (name, logo, description, owner, created date, status), manage members (list, invite, change roles - UI not fully shown but implied), toggle organization active status (with confirmation). Admins/Owners can access the edit page. * **Editing (`edit_organization.html`):** Form to update organization name, billing email, description, and upload/change the logo (with live preview via JS). Uses Django forms and messages framework for validation and feedback. * **Switcher (`components/organization_switcher.html`):** Dropdown component (likely in the main navigation) displaying the current organization (logo/initial, name). Lists other organizations the user belongs to, allowing switching via a form POST that includes the target `organization_id` and current `redirect_url`. 2. **Agent/Crew/Task Configuration & Management (`agents/`):** * **Forms (Detailed Fields):** * **Agent (`AgentForm`, `agent_form.html`):** `name`, `role`, `goal` (textarea), `backstory` (textarea), `llm` (dropdown from `get_models()`), `function_calling_llm` (dropdown), `max_iter`, `max_rpm`, `memory` (boolean switch), `verbose` (boolean switch), `allow_delegation` (boolean switch), `step_callback` (text input), `tools` (checkbox multi-select), `system_template` (textarea), `prompt_template` (textarea), `response_template` (textarea), `allow_Code Execution` (boolean switch), `use_system_prompt` (boolean switch), `respect_context_window` (boolean switch), `max_retry_limit` (number input), `avatar` (radio select with image previews from `AVATAR_CHOICES`). JS enhances form element classes and handles avatar selection UI. * **Crew (`CrewForm`, `crew_form.html`):** `name`, `agents` (multi-select), `tasks` (multi-select), `process` (dropdown: sequential/hierarchical), `manager_llm` (dropdown), `function_calling_llm` (dropdown), `config` (JSON textarea), `max_rpm` (number input), `language` (text input), `language_file` (text input), `embedder` (text input), `output_log_file` (text input), `manager_agent` (dropdown), `manager_callbacks` (text input), `prompt_file` (text input), `planning_llm` (dropdown), `verbose` (boolean switch), `memory` (boolean switch), `cache` (boolean switch), `full_output` (boolean switch), `share_crew` (boolean switch), `planning` (boolean switch). Includes a list (`#task-order-list`) to display/reorder selected tasks. * **Task (`TaskForm`, `task_form.html`):** `description` (textarea), `agent` (dropdown), `expected_output` (textarea), `tools` (checkbox multi-select), `async_execution` (boolean switch), `context` (checkbox multi-select), `config` (JSON textarea), `output_json` (text input), `output_pydantic` (text input), `output_file` (text input), `human_input` (boolean switch), `converter_cls` (text input). * **Tool (`ToolForm`, `tool_form.html`):** `tool_class` (dynamic dropdown), `tool_subclass` (dynamic dropdown populated via AJAX based on `tool_class`), `name` (hidden, populated from subclass), `description` (read-only textarea populated via AJAX). * **Management Views:** Standard Django admin-like views for listing, adding, editing, deleting Agents, Crews, Tasks, and Tools (`manage_agents.html`, `manage_crews.html`, `manage_tasks.html`, `manage_tools.html`). Includes confirmation pages for deletion (`confirm_delete.html`). * **Task Duplication:** Functionality (`duplicate_task` view) to create a copy of an existing task, appending "(Copy)" to the description. 3. **Chat Interface (`agents/chat/`, `chat.html`, `chat.js`):** * **Core Interaction:** Primary UI for interacting with individual agents or crews via WebSockets (`ChatWebsocket` class). * **Participant Selection:** Dropdowns (`#agentSelect`, `#clientSelect`, `#modelSelect`) to choose the agent/crew, associated client, and LLM model (for agents). Selection updates the UI header (`#agent-avatar`, `#agent-name`) via `_updateAgentAvatar`. * **Message Handling (`MessageHandler` class):** Parses incoming WebSocket messages (`assistant_message`, `tool_start`, `tool_result`, `stream`, `human_input_request`, `crew_message`, etc.) and updates the message list (`MessageList` class). Handles Markdown rendering (`markdown-it`), code highlighting (`highlight.js`), and displays tool inputs/outputs. Tracks processed messages (`processedMessageIds`). * **Sending Messages (`_sendMessage`):** Sends user input via WebSocket. Differentiates between: * **Human Input Response:** Sends `user_message` with `context` if `lastHumanInputContext` is set. * **Crew Initialization:** Sends `start_crew` if it's the first message to a selected crew. * **Crew Message:** Sends `crew_message` for subsequent messages to an initialized crew. * **Agent Message:** Sends a standard message with `agent_id`, `model`, `client_id`. * **Message Editing (`editMessage`):** Allows users to click an edit button on their messages. Populates the input field, deletes subsequent messages from the UI (`messageList.deleteMessagesFromIndex`), and sends an `edit` type message via WebSocket with the `message_id`. * **Other Features:** Copy message content (`copyMessage`), new chat creation (`newChatBtn`), conversation deletion (`deleteConversation` with confirmation). 4. **Crew Kanban Board (`agents/templates/crew_kanban.html`, `crew_kanban.js`):** * **Visualization:** Provides a Kanban-style view of ongoing crew executions, likely showing stages/tasks as cards moving through columns (e.g., To Do, In Progress, Done). * **Details Modal (`modals/details_modal.html`):** Clicking on a card/stage opens a modal (`#detailsModal`) displaying detailed information: Status, Stage Type, Agent involved, and the Content (likely input/output/logs) associated with that stage. 5. **Extensive Tool Library (`agents/tools/`):** * (Includes previously listed tools: Web Interaction, Search, SEO, Content Processing, Data Manipulation, Agent Control, External APIs) * **Tool Execution Flow:** Triggered by agents/crews -> `run_tool` Celery task -> Load Tool -> Validate Inputs (Pydantic) -> Set Org Context -> Execute `_run` (sync/async) -> Record `ToolRun` -> Return result/error. 6. **Deep Research Capability (`research/`):** * (As previously detailed: multi-step process, real-time updates via WebSockets, detailed UI showing steps, sources, reasoning, report). 7. **SEO Management Suite (`seo_manager/`):** * **Client Dashboard (`client_detail.html`):** * (As previously detailed: Objectives, Keywords with history charts, Projects, Client Profile). * **Client Profile (Manual):** Integrates Quill editor (`initializeQuillEditors`) within modals (`#add-profile`, `#edit-profile`) for manual creation/editing of client profiles. Form submissions (`addProfileForm`, `editProfileForm`) handle saving via AJAX, showing loading states and using SweetAlert for feedback. * **Client Deletion:** Delete button triggers `handleDeleteClient` JS function, which uses SweetAlert for confirmation (`Are you sure?`) before calling `deleteClientRequest` to make the AJAX POST request to `urls.deleteClient`. Uses SweetAlert for loading and success/error feedback. Redirects to `urls.clientList` on success. * **Analytics & Search Console (`client_analytics.html`, `client_integrations.html`):** * (As previously detailed: OAuth connection management, GA/SC data fetching via services/API clients, date range selection, rendering data in cards and tables using ApexCharts and simpleDatatables.js, error handling including `invalid_grant`). * **Reporting (`keyword_report.html`):** * (As previously detailed: Summary stats, top movers tables). 8. **Content Summarization (`summarizer/summarize.html`):** * **Input:** Accepts text, URL, or YouTube link via textarea (`#query-text`). * **Crawl Option:** Checkbox (`#crawl-website`) enables crawling an entire website (if input is a URL). When checked, reveals a number input (`#max-pages`) to limit crawl depth (default 10, max 100). * **Model Selection:** If user is admin (`request.user.is_superuser`), a multi-select box (`#model-selected`) allows choosing the LLM model. * **Execution:** Form submission triggers AJAX request (likely to `summarize_view`), displays progress (`#task-status` with spinner, timer, progress bar), and shows results in `#task-result`. Uses SweetAlert for feedback during polling/saving. Includes a Cancel button (`#cancel-btn`). * **History:** Sidebar (`#summarization-list`) lists previous summaries. Clicking an item populates the query textarea and result area using JS event listeners. * **Output:** Displays HTML result (`#task-result`) with a copy-to-clipboard button (`#copy-btn`). 9. **File Management (`file_manager/file-manager.html`):** * **UI:** Two-pane layout (directory tree, file list). Uses Bootstrap, FontAwesome. * **Actions:** Upload (button or drag-drop, with SweetAlert loading state via `submitForm` JS), Download (files or folders as zip), Delete (with SweetAlert confirmation via `deleteItem` JS function). * **Previews (Modals):** Generates modals (`#file-{{forloop.counter}}`) for file previews: * Images (`jpg, jpeg, png, gif`): `<img>` tag pointing to `file_manager:preview`. * Videos (`mp4, webm, ogg`): `<video>` tag pointing to `file_manager:preview`. * Text (`txt, log, md, json, etc.`): `<iframe>` pointing to `file_manager:preview`, showing first 50 lines with a download link. * PDF: `<iframe>` pointing to `file_manager:preview`. * CSV: Renders first 10 rows directly in a `<pre>` tag (`{{ item.csv_text }}`) with a download link. * Other: "Preview not available". * **Info Modal (`#info-{{forloop.counter}}`):** Allows viewing and *saving* file information via a form posting to `file_manager:save_info`. Uses `info_value` template tag to display current info. #### Technical Architecture & Stack (Refined) 1. **Backend:** Python/Django, Pydantic (v2 likely, using `model_json_schema`), LangChain, CrewAI (or inspired framework), Celery, Django Channels, Django ORM, `python-json-logger`, Google API Client libraries (GA4 Data API, Search Console API), `requests`, `beautifulsoup4`, `lxml`, `tiktoken`, Embedchain (implied). 2. **Frontend:** Django Templates (DTL), Bootstrap 5, HTMX, Vanilla JS, WebSockets API, ApexCharts (for GA charts), Chart.js (for keyword history/audit charts), Quill (rich text editor), simpleDatatables.js / DataTables.js (table enhancement), Flatpickr (date pickers), SweetAlert2 (notifications/confirmations), Markdown-it (Markdown rendering), Autosize (textarea resizing), Font Awesome (icons), SortableJS (implied for drag-drop lists). 3. **Database:** Relational DB (PostgreSQL/MySQL typical for Django). 4. **Infrastructure:** Web Server (Gunicorn/uWSGI), ASGI Server (Daphne/Uvicorn), Celery Broker (Redis/RabbitMQ), Celery Result Backend (Redis/DB), Web Server (Nginx/Apache). 5. **Key Utilities:** `common.utils.get_llm`, `common.utils.get_models`, `agents.utils` (tool loading/discovery), `organizations.utils.OrganizationContext`, `seo_manager.services` (GA/SC data fetching). #### Overall Capabilities (Reaffirmed) This platform provides a powerful, integrated, and extensible environment for automating complex, multi-step workflows involving AI agents, particularly focused on digital marketing (SEO, research, content) and potentially client management. Its multi-tenant architecture, extensive tool library, asynchronous processing, and real-time feedback mechanisms make it a sophisticated solution for agencies or teams looking to leverage AI automation at scale. The codebase demonstrates a deep integration of various modern web technologies. This revised outline should now cover the breadth and depth of the features and architecture indicated in the provided code snippets much more accurately."