# Extra Configuration Source: https://docs.osmedeus.org/advanced/api-key-configuration Multiple ways to install Osmedeus This guide covers advanced configuration options including binary registry, API keys, storage, LLM, and notifications. ## Configuration File Default location: `~/osmedeus-base/osm-settings.yaml` Override with: `osmedeus --settings-file /path/to/config.yaml` *** ## Adding New Binaries to Registry The binary registry defines tools that can be installed via `osmedeus install binary`. ### Registry Format Create or modify `registry-metadata.json`: ```json theme={null} { "mytool": { "desc": "Description of mytool", "repo_link": "https://github.com/user/mytool", "version": "1.2.0", "tags": ["recon", "scanning"], "valide-command": "mytool --version", "linux": { "amd64": "https://github.com/user/mytool/releases/download/v1.2.0/mytool-linux-amd64.tar.gz", "arm64": "https://github.com/user/mytool/releases/download/v1.2.0/mytool-linux-arm64.tar.gz" }, "darwin": { "amd64": "https://github.com/user/mytool/releases/download/v1.2.0/mytool-darwin-amd64.tar.gz", "arm64": "https://github.com/user/mytool/releases/download/v1.2.0/mytool-darwin-arm64.tar.gz" }, "nix_package": "mytool" } } ``` ### Registry Fields | Field | Description | | --------------------------------- | --------------------------------- | | `desc` | Tool description | | `repo_link` | Repository URL | | `version` | Current version | | `tags` | Categories for filtering | | `valide-command` | Command to verify installation | | `linux`, `darwin`, `windows` | Download URLs per OS/architecture | | `command-linux`, `command-darwin` | Alternative install commands | | `nix_package` | Nix package name | ### Using Custom Registry ```bash theme={null} # Use local registry osmedeus install binary -n mytool -r /path/to/registry.json # Use remote registry osmedeus install binary -n mytool -r https://example.com/registry.json ``` *** ## API Keys Configuration Configure API keys for external services in the `global_vars` section. ### Configuration ```yaml theme={null} global_vars: # GitHub API (for asset discovery) GITHUB_API_KEY: value: "ghp_xxxxxxxxxxxx" as_env: true # Shodan API SHODAN_API_KEY: value: "xxxxxxxxxxxxxxxx" as_env: true # Censys API CENSYS_API_ID: value: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" as_env: true CENSYS_API_SECRET: value: "xxxxxxxxxxxxxxxxxxxx" as_env: true # SecurityTrails SECURITYTRAILS_API_KEY: value: "xxxxxxxxxxxxxxxxxxxx" as_env: true # VirusTotal VIRUSTOTAL_API_KEY: value: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" as_env: true # Custom variables (workflow-only, not exported to env) custom_wordlist: value: "/path/to/wordlist.txt" as_env: false ``` ### Setting via CLI ```bash theme={null} osmedeus config set global_vars.GITHUB_API_KEY ghp_xxxx osmedeus config set global_vars.SHODAN_API_KEY xxxx ``` ### Usage in Workflows ```yaml theme={null} # As template variable - command: 'curl -H "Authorization: token {{GITHUB_API_KEY}}" https://api.github.com/...' # As environment variable (when as_env: true) - command: 'shodan search "hostname:{{Target}}"' # Uses $SHODAN_API_KEY ``` *** ### Usage in Workflows ```yaml theme={null} - name: upload-results type: function function: 'cdnUpload("{{Output}}/results.json", "scans/{{Target}}/results.json")' - name: download-data type: function function: 'cdnDownload("wordlists/common.txt", "{{Output}}/wordlist.txt")' ``` *** ## LLM Configuration Configure AI/LLM providers for the `llm` step type. ### Configuration ```yaml theme={null} llm_config: llm_providers: # Primary provider - provider: ollama base_url: "http://localhost:11434/v1/chat/completions" auth_token: "" model: "llama2" # Fallback provider - provider: openai base_url: "https://api.openai.com/v1/chat/completions" auth_token: "sk-xxxxxxxxxxxx" model: "gpt-4" # LLM settings max_tokens: 1000 temperature: 0.7 top_p: 0.9 max_retries: 3 timeout: 120s stream: false ``` ### Supported Providers | Provider | Base URL | | ------------ | -------------------------------------------- | | Ollama | `http://localhost:11434/v1/chat/completions` | | OpenAI | `https://api.openai.com/v1/chat/completions` | | Azure OpenAI | `https://.openai.azure.com/...` | | Custom | Any OpenAI-compatible endpoint | ### Usage in Workflows ```yaml theme={null} - name: analyze type: llm messages: - role: system content: "You are a security analyst." - role: user content: "Analyze: {{Target}}" exports: analysis: "{{llm_step_content}}" ``` *** ## Database Configuration ### SQLite (Default) ```yaml theme={null} database: db_engine: sqlite db_path: "{{base_folder}}/database-osm.sqlite" ``` ### PostgreSQL ```yaml theme={null} database: db_engine: postgresql host: localhost port: 5432 username: osmedeus password: secure_password db_name: osmedeus connection_timeout: 60 ssl_mode: disable # disable, require, verify-ca, verify-full ``` ### Setting via CLI ```bash theme={null} osmedeus config set database.db_engine postgresql osmedeus config set database.host localhost osmedeus config set database.password mypassword ``` *** ## Server Authentication ### Simple Auth (Username/Password) ```yaml theme={null} server: simple_user_map_key: admin: "secure_password" readonly: "another_password" ``` ### JWT Configuration ```yaml theme={null} server: jwt: secret_signing_key: "change-this-to-random-string" expiration_minutes: 180 ``` ### API Key Authentication ```yaml theme={null} server: enabled_auth_api: true auth_api_key: "your-secure-api-key" ``` When enabled, all API requests require header: `x-osm-api-key: your-secure-api-key` ### Setting via CLI ```bash theme={null} osmedeus config set server.username admin osmedeus config set server.password secure123 osmedeus config set server.jwt.secret_signing_key random_key_here osmedeus config set server.enabled_auth_api true osmedeus config set server.auth_api_key my-api-key ``` *** ## Scan Tactics Configure thread counts for different scan intensities: ```yaml theme={null} scan_tactic: aggressive: 40 # --tactic aggressive default: 10 # default behavior gently: 5 # --tactic gently ``` Usage: ```bash theme={null} osmedeus run -m recon -t example.com --tactic aggressive osmedeus run -m recon -t example.com --tactic gently ``` *** ## Environment Paths ```yaml theme={null} environments: external_binaries_path: "{{base_folder}}/external-binaries" external_data: "{{base_folder}}/external-data" external_configs: "{{base_folder}}/external-configs" workspaces: "{{base_folder}}/workspaces" workflows: "{{base_folder}}/workflows" snapshot: "{{base_folder}}/snapshot" ``` *** ## Quick Setup Checklist 1. **Install binaries:** `osmedeus install binary --all` 2. **Set API keys:** `osmedeus config set global_vars.GITHUB_API_KEY ghp_xxx` 3. **Configure auth:** `osmedeus config set server.password secure123` 4. **Test config:** `osmedeus config list` 5. **Validate setup:** `osmedeus health` # Deployment Source: https://docs.osmedeus.org/advanced/deployment Build, deploy, and run Osmedeus in various environments This guide covers building, deploying, and running Osmedeus in various environments. ## Prerequisites * Go 1.21+ (for local builds) * Docker 20.10+ (for containerized deployment) * Docker Compose 2.0+ (for distributed mode) ## Quick Start ```bash theme={null} # Local build and run make build ./build/bin/osmedeus serve # Docker single container docker build -t osmedeus:latest -f build/docker/Dockerfile . docker run -p 8001:8001 osmedeus:latest # Distributed mode with Docker Compose docker-compose -f build/docker/docker-compose.yml up -d ``` ## Building ### Local Build ```bash theme={null} # Build for current platform make build # Cross-platform builds make build-all # All platforms make build-linux # Linux amd64 make build-darwin # macOS amd64 + arm64 make build-windows # Windows amd64 # Output location ./build/bin/osmedeus ``` ### Docker Build ```bash theme={null} docker build -t osmedeus:latest -f build/docker/Dockerfile . # Development image (with hot-reload) docker build -t osmedeus:dev -f build/docker/Dockerfile.dev . # With custom version docker build --build-arg VERSION=5.1.0 -t osmedeus:5.1.0 -f build/docker/Dockerfile . ``` ## Deployment Modes ### Single Host #### Direct Binary ```bash theme={null} # Run server ./build/bin/osmedeus serve --port 8001 # Run with authentication disabled (development only) ./build/bin/osmedeus serve -A # Run a scan ./build/bin/osmedeus scan -f general -t example.com ``` #### Docker Container ```bash theme={null} # Basic server docker run -d \ --name osmedeus \ -p 8001:8001 \ -v osmedeus-data:/root/osmedeus-base \ -v workspaces:/root/workspaces-osmedeus \ osmedeus:latest # With custom workflows docker run -d \ --name osmedeus \ -p 8001:8001 \ -v /path/to/workflows:/root/osmedeus-base/workflows \ -v /path/to/workspaces:/root/workspaces-osmedeus \ osmedeus:latest ``` ### Distributed Mode (Master/Worker) Distributed mode allows scaling scan workloads across multiple worker nodes using Redis as a message queue. #### Architecture ``` ┌─────────────┐ │ Client │ └──────┬──────┘ │ REST API ┌──────▼──────┐ │ Master │ │ (Server) │ └──────┬──────┘ │ ┌──────▼──────┐ │ Redis │ │ (Queue) │ └──────┬──────┘ ┌────────────┼────────────┐ │ │ │ ┌─────▼────┐ ┌─────▼────┐ ┌─────▼────┐ │ Worker 1 │ │ Worker 2 │ │ Worker N │ └──────────┘ └──────────┘ └──────────┘ ``` #### Docker Compose Setup ```bash theme={null} # Start with 2 workers (default) docker-compose -f build/docker/docker-compose.yml up -d # Scale to 5 workers docker-compose -f build/docker/docker-compose.yml up -d --scale worker=5 # View logs docker-compose -f build/docker/docker-compose.yml logs -f # Stop all services docker-compose -f build/docker/docker-compose.yml down # Stop and remove volumes docker-compose -f build/docker/docker-compose.yml down -v ``` #### Manual Distributed Setup If not using Docker Compose: ```bash theme={null} # 1. Start Redis docker run -d --name redis -p 6379:6379 redis:7-alpine # 2. Start Master ./build/bin/osmedeus serve --master --port 8001 # 3. Start Workers (on same or different machines) ./build/bin/osmedeus worker join --redis-url redis://localhost:6379 ``` #### Submitting Distributed Scans ```bash theme={null} # Submit scan to distributed queue ./build/bin/osmedeus scan -f general -t example.com -D # With custom Redis URL ./build/bin/osmedeus scan -f general -t example.com -D --redis-url redis://redis-host:6379 # Check worker status ./build/bin/osmedeus worker status ``` ## Configuration ### Configuration File Default location: `~/osmedeus-base/osm-settings.yaml` ```yaml theme={null} base_folder: ~/osmedeus-base environments: binaries_path: "{{base_folder}}/binaries" data: "{{base_folder}}/data" workspaces: ~/workspaces-osmedeus workflows: "{{base_folder}}/workflows" server: host: 0.0.0.0 port: 8001 # Required for distributed mode redis: host: localhost port: 6379 password: "" # Optional db: 0 database: db_engine: sqlite # or postgresql db_path: "{{base_folder}}/osm-data.db" client: username: admin password: admin jwt: secret: "change-this-in-production" expiration_minutes: 60 scan_tactic: aggressive: 40 default: 10 gently: 5 ``` ### Environment Variables | Variable | Description | Default | | ----------------- | ---------------- | ---------------- | | `REDIS_HOST` | Redis hostname | localhost | | `REDIS_PORT` | Redis port | 6379 | | `OSM_BASE_FOLDER` | Base folder path | \~/osmedeus-base | ### Command Line Overrides ```bash theme={null} # Override base folder osmedeus -b /custom/path scan -f general -t example.com # Override workflow folder osmedeus -F /custom/workflows workflow list # Override Redis URL (distributed mode) osmedeus scan -f general -t example.com -D --redis-url redis://user:pass@host:6379/0 ``` ## Docker Compose Reference The included `build/docker/docker-compose.yml` provides a complete distributed setup: ### Services | Service | Purpose | Ports | | -------- | ------------------------------- | ----- | | `redis` | Task queue and coordination | 6379 | | `master` | API server and task distributor | 8001 | | `worker` | Task executor (scalable) | - | ### Volumes | Volume | Purpose | | --------------- | --------------------------- | | `redis-data` | Redis persistence | | `osmedeus-data` | Workflows and configuration | | `workspaces` | Scan output data | ### Scaling ```bash theme={null} # Scale workers dynamically docker-compose -f build/docker/docker-compose.yml up -d --scale worker=10 # View running containers docker-compose -f build/docker/docker-compose.yml ps ``` ## Production Considerations ### Security 1. **Authentication**: Never use `-A` (no-auth) in production 2. **JWT Secret**: Change the default JWT secret in config 3. **TLS**: Use a reverse proxy (nginx, traefik) for HTTPS 4. **Network**: Restrict Redis access to internal network only ```yaml theme={null} # Example: Secure JWT configuration client: jwt: secret: "your-256-bit-secret-key-here" expiration_minutes: 30 ``` ### Resource Limits Worker resource limits in docker-compose.yml: ```yaml theme={null} deploy: resources: limits: cpus: '1' memory: 1G reservations: cpus: '0.5' memory: 512M ``` Adjust based on workflow requirements. ### Health Checks The Docker image includes built-in health checks: ```bash theme={null} # Check master health curl http://localhost:8001/health # Check readiness curl http://localhost:8001/health/ready ``` ### Logging ```bash theme={null} # View master logs docker logs osmedeus-master -f # View all worker logs docker-compose -f build/docker/docker-compose.yml logs -f worker # Log levels are controlled by --verbose/-v flag ./build/bin/osmedeus -v serve ``` ### Database Options For production, consider PostgreSQL instead of SQLite: ```yaml theme={null} database: db_engine: postgresql db_host: postgres-host db_port: 5432 db_name: osmedeus db_user: osmedeus db_password: secure-password ``` ### Backup ```bash theme={null} # Backup volumes docker run --rm \ -v osmedeus-data:/data \ -v $(pwd):/backup \ alpine tar czf /backup/osmedeus-backup.tar.gz /data # Backup workspaces docker run --rm \ -v workspaces:/data \ -v $(pwd):/backup \ alpine tar czf /backup/workspaces-backup.tar.gz /data ``` ## Ansible Deployment Deploy Osmedeus on Ubuntu/Debian servers using Ansible. Uses the official install script, SQLite storage, and no Redis -- designed for simple single-host setups. ### Prerequisites * **Control machine**: Ansible 2.12+ * **Target server**: Ubuntu 20.04+ or Debian 11+ * SSH access with root or sudo privileges ### Quick Start ```bash theme={null} cd build/infra # 1. Copy and edit inventory cp inventory.example.ini inventory.ini # Edit inventory.ini with your server IP/hostname # 2. Deploy with secure credentials ansible-playbook -i inventory.ini deploy.yaml \ -e osm_admin_password=YourSecurePassword \ -e osm_jwt_secret=$(openssl rand -base64 32) # 3. Dry run (preview changes without applying) ansible-playbook -i inventory.ini deploy.yaml --check ``` ### What It Does 1. Installs system dependencies (curl, tmux, git, chromium, etc.) 2. Installs Osmedeus via `curl -fsSL https://www.osmedeus.org/install.sh | bash` 3. Deploys `osm-settings.yaml` configured with SQLite (no Redis) 4. Runs `osmedeus health` to verify the installation 5. Sets up a systemd service for auto-start on boot ### Playbook Files ``` build/infra/ ├── deploy.yaml # Main playbook ├── inventory.example.ini # Example inventory └── templates/ ├── osm-settings.yaml.j2 # Settings config template └── osmedeus.service.j2 # Systemd unit template ``` ### Variables | Variable | Default | Description | | ---------------------------- | ----------------------------------- | ----------------------------- | | `osm_server_port` | `8002` | API server port | | `osm_admin_user` | `admin` | Admin username | | `osm_admin_password` | `CHANGE_ME_ADMIN_PASSWORD` | Admin password | | `osm_jwt_secret` | `CHANGE_ME_JWT_SECRET_MIN_32_CHARS` | JWT signing secret | | `osm_jwt_expiration_minutes` | `1440` | Token expiry (24h) | | `osm_threads_aggressive` | `50` | Aggressive scan threads | | `osm_threads_default` | `20` | Default scan threads | | `osm_threads_gently` | `5` | Gentle scan threads | | `osm_enable_service` | `true` | Install systemd service | | `osm_telegram_enabled` | `false` | Enable Telegram notifications | | `osm_telegram_bot_token` | `""` | Telegram bot token | | `osm_telegram_chat_id` | `""` | Telegram chat ID | | `osm_global_variables` | `[]` | Extra env vars for workflows | ### Customization Override any variable at deploy time with `-e`: ```bash theme={null} # Custom port and thread settings ansible-playbook -i inventory.ini deploy.yaml \ -e osm_server_port=9090 \ -e osm_threads_default=30 # With Telegram notifications ansible-playbook -i inventory.ini deploy.yaml \ -e osm_telegram_enabled=true \ -e osm_telegram_bot_token=your_bot_token \ -e osm_telegram_chat_id=your_chat_id # Skip systemd service setup ansible-playbook -i inventory.ini deploy.yaml \ -e osm_enable_service=false ``` ### Post-Deployment ```bash theme={null} # Check service status ssh root@YOUR_SERVER systemctl status osmedeus # Run a scan ssh root@YOUR_SERVER osmedeus run -f general -t example.com # View logs ssh root@YOUR_SERVER journalctl -u osmedeus -f ``` ## Troubleshooting ### Common Issues **Workers not connecting:** ```bash theme={null} # Check Redis connectivity docker exec osmedeus-redis redis-cli ping # Check worker logs docker-compose logs worker ``` **Scans not executing:** ```bash theme={null} # Verify workflow exists ./build/bin/osmedeus workflow list # Check master logs docker logs osmedeus-master ``` **Port conflicts:** ```bash theme={null} # Use different ports docker run -p 8080:8001 osmedeus:latest ``` ### Useful Commands ```bash theme={null} # Environment health check ./build/bin/osmedeus health # Validate workflows ./build/bin/osmedeus workflow validate # Test workflow (dry-run) ./build/bin/osmedeus scan -f general -t example.com --dry-run ``` # Development Source: https://docs.osmedeus.org/advanced/development Development and hacking on Osmedeus This document describes the technical architecture and development practices for Osmedeus. It's intended for developers who want to understand, modify, or extend the codebase. ## Table of Contents * [Project Structure](#project-structure) * [Architecture Overview](#architecture-overview) * [Core Components](#core-components) * [Workflow Engine](#workflow-engine) * [Execution Pipeline](#execution-pipeline) * [Runner System](#runner-system) * [Authentication Middleware](#authentication-middleware) * [Template Engine](#template-engine) * [Function Registry](#function-registry) * [Scheduler System](#scheduler-system) * [Workflow Linter](#workflow-linter) * [Database Layer](#database-layer) * [Testing](#testing) * [Adding New Features](#adding-new-features) * [CLI Shortcuts and Tips](#cli-shortcuts-and-tips) ## Project Structure ``` osmedeus/ ├── cmd/osmedeus/ # Application entry point ├── internal/ # Private packages │ ├── client/ # Remote API client │ ├── config/ # Configuration management │ ├── console/ # Console output capture │ ├── core/ # Core types (Workflow, Step, Trigger, etc.) │ ├── database/ # SQLite/PostgreSQL via Bun ORM │ ├── distributed/ # Distributed execution (master/worker) │ ├── executor/ # Workflow execution engine │ ├── functions/ # Utility functions (Goja JS runtime) │ ├── heuristics/ # Target type detection │ ├── installer/ # Binary installation (direct/Nix) │ ├── linter/ # Workflow linting and validation │ ├── logger/ # Structured logging (Zap) │ ├── parser/ # YAML parsing and caching │ ├── runner/ # Execution environments (host/docker/ssh) │ ├── scheduler/ # Trigger scheduling (cron/event/watch) │ ├── snapshot/ # Workspace export/import │ ├── state/ # Run state export │ ├── template/ # {{Variable}} interpolation engine │ ├── terminal/ # Terminal UI (colors, tables, spinners) │ ├── updater/ # Self-update via GitHub releases │ └── workspace/ # Workspace management ├── lib/ # Shared library utilities ├── pkg/ # Public packages │ ├── cli/ # Cobra CLI commands │ └── server/ # Fiber REST API server │ ├── handlers/ # Request handlers │ └── middleware/ # Auth middleware (JWT, API Key) ├── public/ # Public assets (examples, presets, UI) ├── test/ # Test suites │ ├── e2e/ # E2E CLI tests │ ├── integration/ # Integration tests │ └── testdata/ # Test workflow fixtures ├── docs/ # API documentation └── build/ # Build artifacts and Docker files ``` ## Architecture Overview Osmedeus follows a layered architecture: ``` ┌─────────────────────────────────────────────────────────────┐ │ CLI / API │ │ (pkg/cli, pkg/server) │ ├─────────────────────────────────────────────────────────────┤ │ Executor Layer │ │ ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ │ │ │ Executor │ │ Dispatcher │ │ Step Executors │ │ │ │ │ │ │ │ (bash, function, │ │ │ │ │ │ │ │ foreach, etc.) │ │ │ └─────────────┘ └──────────────┘ └────────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ Runner Layer │ │ ┌──────────────┐ ┌───────────────┐ ┌─────────────────┐ │ │ │ Host Runner │ │ Docker Runner │ │ SSH Runner │ │ │ └──────────────┘ └───────────────┘ └─────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ Support Systems │ │ ┌──────────────┐ ┌───────────────┐ ┌─────────────────┐ │ │ │ Template │ │ Functions │ │ Scheduler │ │ │ │ Engine │ │ Registry │ │ (triggers) │ │ │ └──────────────┘ └───────────────┘ └─────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ Data Layer │ │ ┌──────────────┐ ┌───────────────┐ ┌─────────────────┐ │ │ │ Parser/ │ │ Database │ │ Workspace │ │ │ │ Loader │ │ (SQLite/PG) │ │ Manager │ │ │ └──────────────┘ └───────────────┘ └─────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ``` ## Core Components ### Workflow Types ```go theme={null} // internal/core/workflow.go type Workflow struct { Kind WorkflowKind // "module" or "flow" Name string Description string Params []Param Triggers []Trigger Runner RunnerType RunnerConfig *RunnerConfig Steps []Step // For modules Modules []ModuleRef // For flows } ``` **Module**: Single execution unit with sequential steps **Flow**: Orchestrates multiple modules with dependency management ### Step Types ```go theme={null} // internal/core/step.go type Step struct { Name string Type StepType // bash, function, foreach, parallel-steps, remote-bash, http, llm PreCondition string // Skip condition Command string // For bash/remote-bash Commands []string // Multiple commands Function string // For function type Input string // For foreach Variable string // Foreach variable name Threads int // Foreach parallelism Step *Step // Nested step for foreach ParallelSteps []Step // For parallel-steps type StepRunner RunnerType // For remote-bash: docker or ssh StepRunnerConfig *StepRunnerConfig // Runner config for remote-bash Exports map[string]string OnSuccess []Action OnError []Action Decision *DecisionConfig // Conditional branching (switch/case) } ``` #### remote-bash Step Type The `remote-bash` step type allows per-step Docker or SSH execution, independent of the module-level runner: ```yaml theme={null} steps: - name: docker-scan type: remote-bash step_runner: docker step_runner_config: image: alpine:latest volumes: - /data:/data command: nmap -sV {{target}} - name: ssh-scan type: remote-bash step_runner: ssh step_runner_config: host: "{{ssh_host}}" port: 22 user: "{{ssh_user}}" key_file: ~/.ssh/id_rsa command: whoami && hostname ``` #### Decision Routing (Conditional Branching) Steps can include decision routing to jump to different steps based on switch/case matching: ```yaml theme={null} steps: - name: detect-type type: bash command: echo "{{target_type}}" exports: detected_type: "output" decision: switch: "{{detected_type}}" cases: "domain": goto: subdomain-enum "ip": goto: port-scan "cidr": goto: network-scan default: goto: generic-recon - name: subdomain-enum type: bash command: subfinder -d {{target}} decision: switch: "always" cases: "always": goto: _end # Special value to end workflow ``` The `_end` special value terminates workflow execution from the current step. ### Execution Context ```go theme={null} // internal/core/context.go type ExecutionContext struct { WorkflowName string WorkflowKind WorkflowKind RunID string Target string Variables map[string]interface{} Params map[string]string Exports map[string]interface{} StepIndex int Logger *zap.Logger } ``` The context is passed through the execution pipeline and accumulates state: * Variables are set by the executor (built-in variables) * Params are user-provided * Exports are step outputs that propagate to subsequent steps ## Workflow Engine ### Parser The parser (`internal/parser/parser.go`) handles YAML parsing: ```go theme={null} type Parser struct{} func (p *Parser) Parse(path string) (*core.Workflow, error) func (p *Parser) Validate(workflow *core.Workflow) error ``` ### Loader The loader (`internal/parser/loader.go`) provides caching and lookup: ```go theme={null} type Loader struct { workflowsDir string modulesDir string cache map[string]*core.Workflow } func (l *Loader) LoadWorkflow(name string) (*core.Workflow, error) func (l *Loader) ListFlows() ([]string, error) func (l *Loader) ListModules() ([]string, error) ``` Lookup order: 1. Check cache 2. Try `workflows/.yaml` 3. Try `workflows/-flow.yaml` 4. Try `workflows/modules/.yaml` 5. Try `workflows/modules/-module.yaml` ## Execution Pipeline ### Flow ``` ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ CLI/API │────▶│ Executor │────▶│ Dispatcher │ └──────────────┘ └──────────────┘ └──────────────┘ │ ┌────────────────────────────┼────────────────────────────┐ │ │ │ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ BashExecutor │ │FunctionExec │ │ForeachExec │ └──────────────┘ └──────────────┘ └──────────────┘ ┌──────────────┐ ┌──────────────┐ │ HTTPExecutor │ │ LLMExecutor │ └──────────────┘ └──────────────┘ │ │ │ └────────────────────────────┼────────────────────────────┘ ▼ ┌──────────────┐ │ Runner │ └──────────────┘ ``` ### Executor ```go theme={null} // internal/executor/executor.go type Executor struct { templateEngine *template.Engine functionRegistry *functions.Registry stepDispatcher *StepDispatcher } func (e *Executor) ExecuteModule(ctx context.Context, module *core.Workflow, params map[string]string, cfg *config.Config) (*core.WorkflowResult, error) func (e *Executor) ExecuteFlow(ctx context.Context, flow *core.Workflow, params map[string]string, cfg *config.Config) (*core.WorkflowResult, error) ``` Key responsibilities: 1. Initialize execution context with built-in variables 2. Create and setup the appropriate runner 3. Iterate through steps, dispatching to appropriate handler 4. Handle pre-conditions, exports, and decision routing 5. Process on\_success/on\_error actions ### Step Dispatcher The dispatcher uses a plugin registry pattern for extensible step type handling: ```go theme={null} // internal/executor/dispatcher.go type StepDispatcher struct { registry *PluginRegistry // Extensible executor registry templateEngine *template.Engine functionRegistry *functions.Registry bashExecutor *BashExecutor // Registered as plugin llmExecutor *LLMExecutor // Registered as plugin runner runner.Runner } // PluginRegistry manages step type executors type PluginRegistry struct { executors map[core.StepType]StepExecutor } // StepExecutor interface for all step type handlers type StepExecutor interface { CanHandle(stepType core.StepType) bool Execute(ctx context.Context, step *core.Step, execCtx *core.ExecutionContext, runner runner.Runner) (*core.StepResult, error) } func (d *StepDispatcher) Dispatch(ctx context.Context, step *core.Step, execCtx *core.ExecutionContext) (*core.StepResult, error) ``` Built-in executors registered at startup: * `BashExecutor` - handles `bash` steps * `FunctionExecutor` - handles `function` steps * `ForeachExecutor` - handles `foreach` steps * `ParallelExecutor` - handles `parallel-steps` steps * `RemoteBashExecutor` - handles `remote-bash` steps * `HTTPExecutor` - handles `http` steps * `LLMExecutor` - handles `llm` steps ## Runner System ### Interface ```go theme={null} // internal/runner/runner.go type Runner interface { Execute(ctx context.Context, command string) (*CommandResult, error) Setup(ctx context.Context) error Cleanup(ctx context.Context) error Type() core.RunnerType IsRemote() bool } type CommandResult struct { Output string ExitCode int Error error } ``` ### Host Runner Simple local execution using `os/exec`: ```go theme={null} func (r *HostRunner) Execute(ctx context.Context, command string) (*CommandResult, error) { cmd := exec.CommandContext(ctx, "sh", "-c", command) // ... execute and capture output } ``` ### Docker Runner Supports both ephemeral (`docker run --rm`) and persistent (`docker exec`) modes: ```go theme={null} type DockerRunner struct { config *core.RunnerConfig containerID string // For persistent mode } func (r *DockerRunner) Execute(ctx context.Context, command string) (*CommandResult, error) { if r.config.Persistent && r.containerID != "" { return r.execInContainer(ctx, command) } return r.runEphemeral(ctx, command) } ``` ### SSH Runner Uses `golang.org/x/crypto/ssh` for remote execution: ```go theme={null} type SSHRunner struct { config *core.RunnerConfig client *ssh.Client } func (r *SSHRunner) Setup(ctx context.Context) error { // Build auth methods (key or password) // Establish SSH connection // Optionally copy binary to remote } ``` ## Authentication Middleware ### Auth Types The server supports two authentication methods: | Method | Header | Description | | ------- | ------------------------------- | --------------------------- | | API Key | `x-osm-api-key` | Simple token-based auth | | JWT | `Authorization: Bearer ` | Token from `/osm/api/login` | ### Priority Logic ```go theme={null} // pkg/server/server.go - setupRoutes() if s.config.Server.EnabledAuthAPI { api.Use(middleware.APIKeyAuth(s.config)) } else if !s.options.NoAuth { api.Use(middleware.JWTAuth(s.config)) } ``` Priority order: 1. **API Key Auth** - If `EnabledAuthAPI` is true 2. **JWT Auth** - If API key auth disabled and NoAuth is false 3. **No Auth** - If NoAuth option is true ### APIKeyAuth Implementation ```go theme={null} // pkg/server/middleware/auth.go func APIKeyAuth(cfg *config.Config) fiber.Handler { return func(c *fiber.Ctx) error { apiKey := c.Get("x-osm-api-key") if !isValidAPIKey(apiKey, cfg.Server.AuthAPIKey) { return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{ "error": true, "message": "Invalid or missing API key", }) } return c.Next() } } ``` Security features: * Case-sensitive exact matching * Rejects empty/whitespace-only keys * Rejects placeholder values ("null", "undefined", "nil") ## Template Engine ### Variable Resolution The template engine (`internal/template/engine.go`) handles `{{variable}}` interpolation: ```go theme={null} type Engine struct{} func (e *Engine) Render(template string, ctx map[string]interface{}) (string, error) ``` Resolution order: 1. Check context variables 2. Check environment variables (optional) 3. Return empty string if not found ### Built-in Variable Injection ```go theme={null} // internal/executor/executor.go func (e *Executor) injectBuiltinVariables(cfg *config.Config, params map[string]string, execCtx *core.ExecutionContext) { execCtx.SetVariable("BaseFolder", cfg.BaseFolder) execCtx.SetVariable("Target", params["target"]) execCtx.SetVariable("Output", filepath.Join(workspacesPath, targetSpace)) execCtx.SetVariable("threads", threads) execCtx.SetVariable("RunUUID", execCtx.RunUUID) // ... more variables } ``` ### Foreach Variable Syntax Foreach uses `[[variable]]` syntax (double brackets) to avoid conflicts with template variables: ```yaml theme={null} - name: process-items type: foreach input: "/path/to/items.txt" variable: item step: command: echo [[item]] # Replaced during foreach iteration ``` ## Function Registry ### Otto JavaScript Runtime Functions are implemented in Go and exposed to an Otto JavaScript VM: ```go theme={null} // internal/functions/otto_runtime.go type OttoRuntime struct { vm *otto.Otto } func NewOttoRuntime() *OttoRuntime { vm := otto.New() runtime := &OttoRuntime{vm: vm} runtime.registerFunctions() return runtime } func (r *OttoRuntime) registerFunctions() { r.vm.Set("fileExists", r.fileExists) r.vm.Set("fileLength", r.fileLength) r.vm.Set("trim", r.trim) // ... register all functions } ``` ### Adding New Functions 1. Add the Go implementation in the appropriate file: ```go theme={null} // internal/functions/file_functions.go func (r *OttoRuntime) myNewFunction(call otto.FunctionCall) otto.Value { arg := call.Argument(0).String() // ... implementation result, _ := r.vm.ToValue(output) return result } ``` 2. Register in `registerFunctions()`: ```go theme={null} r.vm.Set("myNewFunction", r.myNewFunction) ``` ### Output and Control Functions These functions provide output and execution control within workflows: ```go theme={null} // internal/functions/util_functions.go // printf prints a message to stdout func (r *OttoRuntime) printf(call otto.FunctionCall) otto.Value // catFile prints file content to stdout func (r *OttoRuntime) catFile(call otto.FunctionCall) otto.Value // exit exits the scan with given code (0=success, non-zero=error) func (r *OttoRuntime) exit(call otto.FunctionCall) otto.Value ``` Usage in workflows: ```yaml theme={null} steps: - name: print-status type: function function: printf("Scan completed for {{Target}}") - name: show-results type: function function: cat_file("{{Output}}/results.txt") ``` ### Event Functions These functions enable event-driven workflows by generating and emitting events: ```go theme={null} // internal/functions/event_functions.go // generate_event emits a single structured event // Usage: generate_event(workspace, topic, source, data_type, data) func (vf *vmFunc) generateEvent(call goja.FunctionCall) goja.Value // generate_event_from_file emits an event for each line in a file // Usage: generate_event_from_file(workspace, topic, source, data_type, filePath) func (vf *vmFunc) generateEventFromFile(call goja.FunctionCall) goja.Value ``` Usage in workflows: ```yaml theme={null} steps: - name: emit-single-event type: function function: | generate_event("{{Workspace}}", "assets.new", "scanner", "subdomain", "api.example.com") - name: emit-from-file type: function function: | generate_event_from_file("{{Workspace}}", "assets.new", "recon", "subdomain", "{{Output}}/subdomains.txt") ``` Event delivery uses a fallback chain: 1. **Server API** - POST to `/osm/api/events/emit` if server configured 2. **Redis Pub/Sub** - Publish to `osm:events:{topic}` in distributed mode 3. **Database Queue** - Store in `event_logs` table with `processed=false` 4. **Webhooks** - Send to configured webhook endpoints ### Function Execution ```go theme={null} // internal/functions/registry.go func (r *Registry) Execute(expr string, ctx map[string]interface{}) (interface{}, error) { return r.runtime.Execute(expr, ctx) } func (r *Registry) EvaluateCondition(condition string, ctx map[string]interface{}) (bool, error) { return r.runtime.EvaluateCondition(condition, ctx) } ``` ## Scheduler System ### Trigger Types ```go theme={null} // internal/core/trigger.go type TriggerType string const ( TriggerManual TriggerType = "manual" TriggerCron TriggerType = "cron" TriggerEvent TriggerType = "event" TriggerWatch TriggerType = "watch" ) ``` ### Scheduler The scheduler manages workflow triggers using gocron for cron jobs and fsnotify for file watching: ```go theme={null} // internal/scheduler/scheduler.go type Scheduler struct { scheduler gocron.Scheduler triggers map[string]*RegisteredTrigger handlers map[string]TriggerHandler events chan *core.Event // File watcher (fsnotify-based) watcher *fsnotify.Watcher watchPaths map[string][]*RegisteredTrigger // path → triggers mapping } func (s *Scheduler) RegisterTrigger(workflow *core.Workflow, trigger *core.Trigger) error func (s *Scheduler) EmitEvent(event *core.Event) error func (s *Scheduler) Start() error // Starts cron scheduler, file watcher, and event listener func (s *Scheduler) Stop() error // Stops all and closes watcher ``` File watching uses fsnotify for instant inotify-based notifications (sub-millisecond latency) instead of polling. ### Event Filtering Events are matched using JavaScript expressions: ```go theme={null} func (s *Scheduler) evaluateFilters(filters []string, event *core.Event) bool { vm := otto.New() vm.Set("event", eventObj) for _, filter := range filters { result, _ := vm.Run(filter) if !result.ToBoolean() { return false } } return true } ``` ## Workflow Linter The workflow linter (`internal/linter/`) provides static analysis of workflow YAML files to catch common issues before execution. ### Usage ```bash theme={null} # Lint a single workflow osmedeus workflow lint my-workflow.yaml # Lint by workflow name (searches in workflows path) osmedeus workflow lint my-workflow # Lint all workflows in a directory osmedeus workflow lint /path/to/workflows/ # Output formats osmedeus workflow lint my-workflow.yaml --format pretty # Default, colored output osmedeus workflow lint my-workflow.yaml --format json # Machine-readable JSON osmedeus workflow lint my-workflow.yaml --format github # GitHub Actions annotations # Filter by severity osmedeus workflow lint my-workflow.yaml --severity warning # Show warnings and above osmedeus workflow lint my-workflow.yaml --severity error # Show only errors # Disable specific rules osmedeus workflow lint my-workflow.yaml --disable unused-variable,empty-step # CI mode (exit with error code if issues found) osmedeus workflow lint my-workflow.yaml --check ``` ### Severity Levels | Severity | Description | Exit Code | | ----------- | ------------------------------------------------ | ---------------- | | **info** | Best practice suggestions (e.g., unused exports) | 0 | | **warning** | Potential issues that may cause problems | 0 | | **error** | Critical issues that will likely cause failures | 1 (with --check) | ### Built-in Rules | Rule | Severity | Description | | ------------------------ | -------- | ------------------------------------------------------ | | `missing-required-field` | warning | Detects missing required fields (name, kind, type) | | `duplicate-step-name` | warning | Detects multiple steps with the same name | | `empty-step` | warning | Detects steps with no executable content | | `unused-variable` | info | Detects exports that are never referenced | | `invalid-goto` | warning | Detects decision goto references to non-existent steps | | `invalid-depends-on` | warning | Detects depends\_on references to non-existent steps | | `circular-dependency` | warning | Detects circular references in step dependencies | **Note**: The `undefined-variable` rule is available but not enabled by default as it can produce false positives for dynamically-injected variables. ### Built-in Variables The linter recognizes all runtime-injected variables to avoid false positives. These include: **Path Variables**: `BaseFolder`, `Binaries`, `Data`, `ExternalData`, `ExternalConfigs`, `Workflows`, `Workspaces`, etc. **Target Variables**: `Target`, `target`, `TargetFile`, `TargetSpace` **Output Variables**: `Output`, `output`, `Workspace`, `workspace` **Metadata Variables**: `Version`, `RunUUID`, `TaskDate`, `TimeStamp`, `Today`, `RandomString` **Heuristic Variables**: `TargetType`, `TargetRootDomain`, `TargetTLD`, `Org`, `TargetHost`, `TargetPort`, etc. **Chunk Variables**: `ChunkIndex`, `ChunkSize`, `TotalChunks`, `ChunkStart`, `ChunkEnd` ### Linter Architecture ```go theme={null} // internal/linter/linter.go type Linter struct { rules []LinterRule options LinterOptions } // LinterRule interface for all lint rules type LinterRule interface { Name() string Description() string Severity() Severity Check(ast *WorkflowAST) []LintIssue } func (l *Linter) Lint(path string) (*LintResult, error) func (l *Linter) LintContent(content []byte, filename string) (*LintResult, error) ``` ### Adding a New Lint Rule 1. Create the rule in `internal/linter/rules.go`: ```go theme={null} type MyNewRule struct{} func (r *MyNewRule) Name() string { return "my-new-rule" } func (r *MyNewRule) Description() string { return "Detects my issue" } func (r *MyNewRule) Severity() Severity { return SeverityWarning } func (r *MyNewRule) Check(wast *WorkflowAST) []LintIssue { var issues []LintIssue // ... implementation return issues } ``` 2. Register in `GetDefaultRules()`: ```go theme={null} func GetDefaultRules() []LinterRule { return []LinterRule{ // ... existing rules &MyNewRule{}, } } ``` ## Database Layer ### Multi-Engine Support ```go theme={null} // internal/database/database.go func Connect(cfg *config.Config) (*bun.DB, error) { switch { case cfg.IsPostgres(): return connectPostgres(cfg) case cfg.IsSQLite(): return connectSQLite(cfg) default: return nil, fmt.Errorf("unsupported database engine") } } ``` ### Models ```go theme={null} // internal/database/models.go type Run struct { ID string RunID string WorkflowName string WorkflowKind string // "flow" or "module" Target string Params map[string]string Status string // "pending", "running", "completed", "failed" Workspace string // Logical workspace name (same as TargetSpace) StartedAt time.Time CompletedAt time.Time ErrorMessage string ScheduleID string TriggerType string // "manual", "cron", "event", "api" TriggerName string TotalSteps int CompletedSteps int CurrentPid int // PID of running process for cancellation RunPriority int // Execution priority RunMode string // "sequential", "parallel", etc. CreatedAt time.Time UpdatedAt time.Time } type Asset struct { ID int64 Workspace string AssetValue string // Primary identifier (hostname) URL string Input string Scheme string // "http", "https" Method string Path string StatusCode int ContentType string ContentLength int64 Title string Words int Lines int HostIP string A []string // DNS A records (JSON) TLS string AssetType string Tech []string // Technologies (JSON) Time string // Response time Remarks string // Labels Source string // Discovery source CreatedAt time.Time UpdatedAt time.Time } type Workspace struct { ID int64 Name string LocalPath string TotalAssets int TotalSubdomains int TotalURLs int TotalVulns int VulnCritical int VulnHigh int VulnMedium int VulnLow int VulnPotential int RiskScore float64 Tags []string // JSON array LastRun time.Time RunWorkflow string StateExecutionLog string // Path to execution log StateCompletedFile string // Path to completed marker file StateWorkflowFile string // Path to workflow state file StateWorkflowFolder string // Path to workflow state folder CreatedAt time.Time UpdatedAt time.Time } type EventLog struct { ID int64 Topic string // "run.started", "run.completed", "asset.discovered", etc. EventID string Name string Source string // "executor", "scheduler", "api" SourceType string // "executor", "scheduler", "api", "trigger" DataType string Data string // JSON payload Workspace string RunID string WorkflowName string Processed bool ProcessedAt time.Time Error string CreatedAt time.Time } type Schedule struct { ID string Name string WorkflowName string WorkflowPath string TriggerName string TriggerType string // "cron", "event", "watch" Schedule string // Cron expression EventTopic string WatchPath string InputConfig map[string]string // JSON params IsEnabled bool LastRun time.Time NextRun time.Time RunCount int CreatedAt time.Time UpdatedAt time.Time } ``` ### Repository Pattern ```go theme={null} // internal/database/repository/asset_repo.go type AssetRepository struct { db *bun.DB } func (r *AssetRepository) Create(ctx context.Context, asset *database.Asset) error func (r *AssetRepository) Search(ctx context.Context, query AssetQuery) ([]*database.Asset, int, error) func (r *AssetRepository) Upsert(ctx context.Context, asset *database.Asset) error ``` ### Schedule Operations ```go theme={null} // internal/database/seed.go func ListSchedules(ctx context.Context, offset, limit int) (*ScheduleResult, error) func GetScheduleByID(ctx context.Context, id string) (*Schedule, error) func CreateSchedule(ctx context.Context, input CreateScheduleInput) (*Schedule, error) func UpdateSchedule(ctx context.Context, id string, input UpdateScheduleInput) (*Schedule, error) func DeleteSchedule(ctx context.Context, id string) error func UpdateScheduleLastRun(ctx context.Context, id string) error ``` ### JSONL Import ```go theme={null} // internal/database/jsonl.go type JSONLImporter struct { db *bun.DB batchSize int } func (i *JSONLImporter) ImportAssets(ctx context.Context, filePath, workspace, source string) (*ImportResult, error) ``` ## Testing ### Test Structure ``` internal/functions/registry_test.go # Function unit tests internal/parser/loader_test.go # Parser/loader unit tests internal/runner/runner_test.go # Runner unit tests internal/executor/executor_test.go # Executor unit tests internal/scheduler/scheduler_test.go # Scheduler unit tests pkg/server/handlers/handlers_test.go # API handler unit tests test/integration/workflow_test.go # Workflow integration tests test/e2e/ # E2E CLI tests ├── e2e_test.go # Common test helpers ├── version_test.go # Version command tests ├── health_test.go # Health command tests ├── workflow_test.go # Workflow command tests ├── function_test.go # Function command tests ├── scan_test.go # Scan command tests ├── server_test.go # Server command tests ├── worker_test.go # Worker command tests ├── distributed_test.go # Distributed scan e2e tests ├── ssh_test.go # SSH runner e2e tests (module & step level) └── api_test.go # API endpoint e2e tests (all routes) ``` ### Running Tests ```bash theme={null} # All unit tests (fast, no external dependencies) make test-unit # Integration tests (requires Docker) make test-integration # E2E CLI tests (requires binary build) make test-e2e # SSH E2E tests - full workflow tests with SSH runner # Tests both module-level (runner: ssh) and step-level (step_runner: ssh) # Uses linuxserver/openssh-server Docker container make test-e2e-ssh # API E2E tests - tests all API endpoints # Starts Redis, seeds database, starts server, tests all routes make test-e2e-api # Distributed scan e2e tests (requires Docker for Redis) make test-distributed # Docker runner tests make test-docker # SSH runner unit tests (using linuxserver/openssh-server) make test-ssh # All tests with coverage make test-coverage ``` ### Writing Tests Use testify for assertions: ```go theme={null} func TestMyFeature(t *testing.T) { // Arrange tmpDir := t.TempDir() // Act result, err := myFunction(tmpDir) // Assert require.NoError(t, err) assert.Equal(t, expected, result) } ``` For integration tests, use build tags: ```go theme={null} func TestDockerRunner_Integration(t *testing.T) { if testing.Short() { t.Skip("skipping integration test") } // ... } ``` ## Adding New Features ### Adding a New Step Type 1. Define the type in `internal/core/types.go`: ```go theme={null} const StepTypeMyNew StepType = "mynew" ``` 2. Create executor in `internal/executor/mynew_executor.go`: ```go theme={null} type MyNewExecutor struct { templateEngine *template.Engine } func (e *MyNewExecutor) Execute(ctx context.Context, step *core.Step, execCtx *core.ExecutionContext) (*core.StepResult, error) { // Implementation } ``` 3. Register in dispatcher (`internal/executor/dispatcher.go`): ```go theme={null} func (d *StepDispatcher) Dispatch(...) (*core.StepResult, error) { switch step.Type { // ... case core.StepTypeMyNew: return d.myNewExecutor.Execute(ctx, step, execCtx) } } ``` ### Adding a New Runner 1. Create runner in `internal/runner/myrunner.go`: ```go theme={null} type MyRunner struct { config *core.RunnerConfig } func (r *MyRunner) Execute(ctx context.Context, command string) (*CommandResult, error) func (r *MyRunner) Setup(ctx context.Context) error func (r *MyRunner) Cleanup(ctx context.Context) error func (r *MyRunner) Type() core.RunnerType func (r *MyRunner) IsRemote() bool ``` 2. Add type in `internal/core/types.go`: ```go theme={null} const RunnerTypeMy RunnerType = "myrunner" ``` 3. Register in factory (`internal/runner/runner.go`): ```go theme={null} func NewRunnerFromType(runnerType core.RunnerType, ...) (Runner, error) { switch runnerType { case core.RunnerTypeMy: return NewMyRunner(config, binaryPath) } } ``` ### Adding a New Installer Mode 1. Create installer in `internal/installer/mymode.go`: ```go theme={null} func InstallBinaryViaMyMode(name, pkg, binariesFolder string) error { // Implementation } ``` 2. Add flag in `pkg/cli/install.go`: ```go theme={null} installBinaryCmd.Flags().BoolVar(&myModeInstall, "my-mode-install", false, "use MyMode to install") ``` 3. Register in `runInstallBinary()` switch statement. See `internal/installer/nix.go` for a complete example. ### Adding a New API Endpoint 1. Add handler in `pkg/server/handlers/handlers.go`: ```go theme={null} func MyHandler(cfg *config.Config) fiber.Handler { return func(c *fiber.Ctx) error { // Implementation return c.JSON(fiber.Map{"data": result}) } } ``` 2. Register route in `pkg/server/server.go`: ```go theme={null} func (s *Server) setupRoutes() { // ... api.Get("/my-endpoint", handlers.MyHandler(s.config)) } ``` ### Adding a New CLI Command 1. Create command file in `pkg/cli/mycommand.go`: ```go theme={null} var myCmd = &cobra.Command{ Use: "mycommand", Short: "Description", RunE: func(cmd *cobra.Command, args []string) error { // Implementation }, } func init() { myCmd.Flags().StringVarP(&myFlag, "flag", "f", "", "description") } ``` 2. Register in `pkg/cli/root.go`: ```go theme={null} func init() { rootCmd.AddCommand(myCmd) } ``` ## CLI Shortcuts and Tips ### Command Aliases * `osmedeus func` - alias for `osmedeus function` * `osmedeus func e` - alias for `osmedeus function eval` * `osmedeus db ls` - alias for `osmedeus db list` ### Database CLI Commands Query and manage database tables directly from the CLI: ```bash theme={null} # List all tables with row counts osmedeus db list # Query specific table (default columns shown) osmedeus db list --table event_logs # List available columns for a table osmedeus db list --table event_logs --list-columns # Filter by specific columns osmedeus db list --table event_logs --columns topic,source,data_type,data # Show all columns including hidden ones (id, timestamps) osmedeus db list --table event_logs --all # Filter by field value osmedeus db list --table event_logs --where topic=assets.new osmedeus db list --table event_logs --where processed=false # Search across all columns osmedeus db list --table event_logs --search "nuclei" # Output as JSON for scripting osmedeus db list --table event_logs --json # Pagination osmedeus db list --table event_logs --offset 50 --limit 100 ``` Default columns per table: * `runs`: run\_uuid, workflow\_name, target, workspace, status, completed\_steps, total\_steps, started\_at * `event_logs`: topic, source, source\_type, processed, data\_type, workspace, data * `artifacts`: artifact\_path, artifact\_type, content\_type, workspace, run\_id * `assets`: asset\_value, host\_ip, title, status\_code, last\_seen\_at, technologies * `schedules`: name, workflow\_name, trigger\_type, schedule, is\_enabled, run\_count * `workspaces`: name, local\_path, total\_assets, total\_vulns, risk\_score, last\_run ### Function Evaluation CLI Evaluate utility functions from the command line with bulk processing support: ```bash theme={null} # Single expression evaluation osmedeus func eval 'log_info("hello")' osmedeus func eval -e 'fileLength("/path/to/file.txt")' # With target variable osmedeus func eval -e 'httpGet("https://" + target)' -t example.com # Bulk processing from file (target variable available in script) osmedeus func eval -e 'log_info("Processing: " + target)' -T targets.txt # Bulk processing with concurrency osmedeus func eval -e 'httpGet("https://" + target)' -T targets.txt -c 10 # Using function files for reusable logic osmedeus func eval --function-file check-host.js -T targets.txt -c 5 # Additional parameters osmedeus func eval -e 'log_info(target + " in " + ws)' -T targets.txt --params ws=production # Function name with arguments osmedeus func eval log_info "hello world" osmedeus func eval -f httpGet "https://example.com" # Read script from stdin echo 'log_info("hello")' | osmedeus func eval --stdin # List available functions osmedeus func list osmedeus func list event # Filter by category ``` ### New Scan Flags * `-c, --concurrency` - Number of targets to scan concurrently * `--timeout` - Scan timeout (e.g., `2h`, `3h`, `1d`) * `--repeat` - Repeat scan after completion * `--repeat-wait-time` - Wait time between repeats (e.g., `30m`, `1h`, `1d`) * `-m` can be specified multiple times to run modules in sequence ### Debugging Tips * Use `osmedeus --usage-example` to see comprehensive examples for all commands * Use `--verbose` or `--debug` for detailed logging * Use `--dry-run` to preview scan execution without running commands * Use `--log-file-tmp` to create timestamped log files for debugging ## Code Style * Use `go fmt` and `golangci-lint` * Follow Go naming conventions * Use structured logging with zap * Return errors, don't panic * Use context for cancellation * Write tests for new features ## Useful Commands ```bash theme={null} # Build make build # Test make test-unit # Format make fmt # Lint make lint # Tidy dependencies make tidy # Generate (if needed) make generate # Generate Swagger docs make swagger # Update embedded UI from dashboard build make update-ui # Install to $GOBIN make install # Docker Toolbox (all tools pre-installed) make docker-toolbox # Build toolbox image make docker-toolbox-run # Start toolbox container make docker-toolbox-shell # Enter container shell ``` # Distributed Execution Source: https://docs.osmedeus.org/advanced/distributed Distributed Execution Scale scanning across multiple machines using Redis coordination. ## Architecture ``` +-----------------+ +-----------------+ | Master | | Redis | | (API Server) |---->| (Task Queue) | +-----------------+ +--------+--------+ | +------------------------+------------------------+ | | | v v v +---------------+ +---------------+ +---------------+ | Worker 1 | | Worker 2 | | Worker N | +---------------+ +---------------+ +---------------+ ``` ## Components ### Master Node * API server for submitting tasks * Task distribution coordinator * Result aggregation * Worker health monitoring ### Workers * Execute assigned tasks * Report progress and results * Auto-reconnect on failure * Heartbeat to master * Worker ID format: `wosm-` (e.g. `wosm-a1b2c3d4`) * Default alias: `wosm-` or `wosm-` when no `--alias` is provided ### Redis * Task queue storage * Worker registration * Result storage * Pub/Sub for events ## Setup ### 0. Start Redis if you don't have one ```bash theme={null} # Docker docker run -d -p 6379:6379 redis:7-alpine # With persistence docker run -d -p 6379:6379 \ -v redis-data:/data \ redis:7-alpine redis-server --appendonly yes # then test connection at localhost:6379 ``` ### 1. Start Master Node ```bash theme={null} ## Configure Redis connection osmedeus config set redis.host 127.0.0.1 osmedeus config set redis.port 6379 osmedeus server --master ``` Or with custom Redis: ```bash theme={null} osmedeus server --master --redis-url redis://redis-host:6379 ``` You can also run the worker node without the master mode with REST server API, provided they can connect to the same Redis instance. However, the results won't be aggregated in the master API and you have to run the scan manually via CLI. ### 2. Start Workers Node ```bash theme={null} osmedeus worker join --redis-url redis://host.docker.internal:6379 ``` With public IP detection (used for default alias and SSH routing): ```bash theme={null} osmedeus worker join --redis-url redis://host.docker.internal:6379 --get-public-ip ``` With a custom alias: ```bash theme={null} osmedeus worker join --redis-url redis://host.docker.internal:6379 --alias my-worker ``` Or if master is on localhost: ```bash theme={null} ## Configure Redis connection osmedeus config set redis.host 127.0.0.1 osmedeus config set redis.port 6379 osmedeus worker join ``` ### 3. Submit Tasks & Listing Workers ```bash theme={null} # Via CLI osmedeus run --distributed-run -f general -t hackerone.com # Via API curl -X POST http://master:8002/osm/api/runs \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"flow": "general", "target": "example.com", "distributed": true}' # List workers (use --json for json output) osmedeus worker ls ``` ### 4. Running Utility Scripts on Master/Worker Nodes (Optional) ```bash theme={null} # run this on the master node to execute the command on all worker nodes osmedeus eval "run_on_worker('all', 'bash', 'touch /tmp/on-worker')" --redis-url redis://host.docker.internal:6379 ## run this on the worker node to execute the command on the master node osmedeus worker --redis-url redis://host.docker.internal:6379 eval "run_on_master('bash', 'touch /tmp/on-master')" ``` *** ## Task Distribution ### Task Lifecycle ``` 1. Client submits task to master 2. Master queues task in Redis 3. Available worker claims task 4. Worker executes workflow 5. Worker reports progress/results 6. Master aggregates results 7. Results available via API ``` ### Load Balancing Tasks are distributed using a pull model: * Workers poll for available tasks * First available worker claims task * No central scheduling required ### Task Priority (Future feature) Tasks can have priority levels: * High: Security-critical scans * Normal: Regular assessments * Low: Background enumeration ## Monitoring ### Worker Status ```bash theme={null} # CLI osmedeus worker status # API curl http://master:8002/osm/api/workers \ -H "Authorization: Bearer $TOKEN" ``` ### Task Status ```bash theme={null} # List tasks curl http://master:8002/osm/api/tasks \ -H "Authorization: Bearer $TOKEN" # Get task details curl http://master:8002/osm/api/tasks/task-123 \ -H "Authorization: Bearer $TOKEN" ``` ## Remote Monitoring Use the `client` command to monitor distributed runs from any machine: ```bash theme={null} # Set connection details export OSM_REMOTE_URL=http://master:8080 export OSM_REMOTE_AUTH_KEY=your-api-key # Monitor runs osmedeus client fetch -t runs --status running --refresh 5s # Check step results for a specific run osmedeus client fetch -t step_results --workspace example_com # View vulnerabilities osmedeus client fetch -t vulnerabilities --severity critical ``` ## Docker Compose Setup ```yaml theme={null} services: redis: image: redis:7-alpine container_name: osm-e2e-redis ports: - "6399:6379" command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 5s timeout: 3s retries: 10 networks: - osm-e2e-network master: build: context: ../.. dockerfile: build/docker/Dockerfile image: osmedeus:e2e container_name: osm-e2e-master ports: - "8002:8002" volumes: - workspaces:/root/workspaces-osmedeus depends_on: redis: condition: service_healthy command: ["serve", "--master", "--redis-url", "redis://redis:6379", "-A"] healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8002/health"] interval: 10s timeout: 5s start_period: 10s retries: 5 networks: - osm-e2e-network worker: image: osmedeus:e2e container_name: osm-e2e-worker volumes: - workspaces:/root/workspaces-osmedeus depends_on: redis: condition: service_healthy master: condition: service_healthy command: ["worker", "join", "--redis-url", "redis://redis:6379"] networks: - osm-e2e-network volumes: workspaces: driver: local networks: osm-e2e-network: driver: bridge ``` Run: ```bash theme={null} # Distributed E2E stack: Redis + master + worker # Usage: make distributed-e2e-up # Build image + start stack + wait for health make distributed-e2e-run # Submit scan + tail worker logs make distributed-e2e-down # Tear down everything ``` # Event-Driven Triggers Source: https://docs.osmedeus.org/advanced/event-driven Build reactive automation pipelines with event-driven workflows Build reactive automation pipelines that respond to discoveries, chain workflows together, and integrate with external systems. ## Overview Example on Trigger Event Event-driven triggers enable workflows to execute automatically in response to events: * **React to discoveries**: Scan new subdomains as they're found * **Chain workflows**: Connect reconnaissance → probing → scanning * **External integration**: Receive webhooks from GitHub, CI/CD, or custom tools * **Real-time automation**: Process findings as they occur ## Event Architecture ### Event Structure Events carry structured data through the system: ```go theme={null} type Event struct { Topic string // Category: "assets.new", "vulnerabilities.new" ID string // Unique event identifier (UUID) Name string // Event name: "vulnerability.discovered" Source string // Origin: "nuclei", "httpx", "amass" Data string // JSON payload DataType string // Type: "subdomain", "url", "finding" Workspace string // Workspace identifier RunID string // Run ID that generated this event WorkflowName string // Workflow that generated this event Timestamp time.Time // When the event occurred } ``` ### Event Flow ``` ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Event Source │ │ Event Queue │ │ Scheduler │ │ │────▶│ (1000 buffer) │────▶│ (Filter + │ │ - Workflows │ │ │ │ Dispatch) │ │ - Functions │ │ Backpressure: │ │ │ │ - Webhooks │ │ 5s timeout │ │ │ └──────────────────┘ └──────────────────┘ └────────┬─────────┘ │ ┌─────────────────────────────────┼─────────────────────────────────┐ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │ Workflow A │ │ Workflow B │ │ Workflow C │ │ (topic match) │ │ (filtered) │ │ (all events) │ └────────────────┘ └────────────────┘ └────────────────┘ ``` ### Backpressure Handling The event system protects against overload: | Parameter | Value | Description | | ---------- | --------- | ------------------------------------ | | Queue Size | 1000 | Maximum events buffered | | Timeout | 5 seconds | Wait time when queue is full | | Behavior | Drop | Events dropped if queue remains full | Monitor queue health via metrics (see [Monitoring Events](#monitoring-events)). ## Emitting Events ### From Workflow Functions Use the `generate_event` function to emit events from workflows: ```yaml theme={null} name: subdomain-discovery kind: module steps: - name: enumerate type: bash command: amass enum -d {{target}} -o {{Output}}/subdomains.txt - name: emit-discoveries type: function function: | generate_event_from_file("{{Workspace}}", "assets.new", "amass", "subdomain", "{{Output}}/subdomains.txt") ``` #### generate\_event(workspace, topic, source, data\_type, data) Emit a single structured event. ```javascript theme={null} // Simple string data generate_event("{{Workspace}}", "assets.new", "httpx", "url", "https://api.example.com") // Complex object data generate_event("{{Workspace}}", "vulnerabilities.new", "nuclei", "finding", { url: "https://example.com/admin", severity: "critical", template: "CVE-2024-1234", matched: "/admin" }) ``` **Parameters:** * `workspace` - Workspace identifier for the event * `topic` - Event topic/category (e.g., "assets.new") * `source` - Event source (e.g., "nuclei", "httpx") * `data_type` - Type of data (e.g., "subdomain", "url", "finding") * `data` - Event payload (string or object) **Returns:** `boolean` - `true` if event was sent successfully #### generate\_event\_from\_file(workspace, topic, source, data\_type, path) Emit an event for each non-empty line in a file. ```javascript theme={null} generate_event_from_file("{{Workspace}}", "assets.new", "subfinder", "subdomain", "{{Output}}/subdomains.txt") // Returns: 42 (number of events generated) ``` **Parameters:** * `workspace` - Workspace identifier for the events * `topic` - Event topic/category * `source` - Event source * `data_type` - Type of data * `path` - File path containing data (one item per line) **Returns:** `integer` - count of events successfully generated ### Webhook Functions Send events to external webhook endpoints configured in settings: ```yaml theme={null} - name: notify-finding type: function function: | notify_webhook("Critical vulnerability found on {{target}}") ``` #### notify\_webhook(message) Send a plain text message to all configured webhooks. ```javascript theme={null} notify_webhook("Scan completed for example.com with 15 findings") ``` #### send\_webhook\_event(eventType, data) Send a structured event to all configured webhooks. ```javascript theme={null} send_webhook_event("scan_complete", { target: "example.com", findings: 15, duration: "2h30m" }) ``` ### From External Systems (Webhooks) Receive webhooks via the API to trigger workflows: ```bash theme={null} curl -X POST http://localhost:8080/osm/api/events/emit \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "topic": "external.webhook", "source": "github", "data_type": "push", "data": "{\"repository\": \"myorg/myapp\", \"branch\": \"main\"}" }' ``` ## Event Triggers ### Trigger Configuration Configure event triggers in workflow YAML: ```yaml theme={null} name: probe-new-assets kind: module triggers: - name: on-subdomain-discovered on: event event: topic: "assets.new" filters: - "event.source == 'amass'" - "event.data_type == 'subdomain'" dedupe_key: "{{event.data.value}}" dedupe_window: "5m" input: type: event_data field: "value" name: target enabled: true params: - name: target required: true steps: - name: probe type: bash command: httpx -u {{target}} -o {{Output}}/httpx.json -json ``` ### EventConfig Fields | Field | Type | Description | | ------------------ | --------- | ------------------------------------------------------------------------- | | `topic` | string | Event topic to match (supports glob patterns: `*`, `?`, `[`; empty = all) | | `filters` | \[]string | JavaScript expressions (all must pass) | | `filter_functions` | \[]string | JavaScript expressions with utility functions available (all must pass) | | `dedupe_key` | string | Template for deduplication key | | `dedupe_window` | string | Time window for deduplication (e.g., "5m") | ### TriggerInput Options Trigger input supports two syntaxes: a legacy syntax and a new exports-style syntax. #### New Exports-Style Syntax (Recommended) Map multiple event fields to workflow variables using a concise syntax: ```yaml theme={null} input: # variable_name: expression target: event_data.url asset_type: event_data.type source: event.source description: trim(event_data.desc) ``` **Expression Types:** * `event_data.` - Access parsed event data fields (e.g., `event_data.url`, `event_data.severity`) * `event.` - Access event metadata (`event.topic`, `event.source`, `event.name`, `event.id`, `event.data_type`, `event.workspace`, `event.run_uuid`, `event.workflow_name`) * `function(...)` - Transform values using utility functions (e.g., `trim(event_data.desc)`, `lower(event_data.name)`) #### Legacy Syntax The original input configuration (still supported for backward compatibility): | Type | Description | Fields | | ------------ | -------------------------- | ------------------------------------- | | `event_data` | Extract from event payload | `field` - JSON path to extract | | `function` | Transform with function | `function` - JS expression (e.g., jq) | | `param` | Static parameter | `name` - parameter name | | `file` | Read from file | `path` - file path | ```yaml theme={null} # Extract field from event data input: type: event_data field: "url" name: target # Transform with jq function input: type: function function: 'jq("{{event.data}}", ".target.url")' name: target # Use static parameter input: type: param name: default_target ``` ### Topic Glob Patterns Event topics support glob patterns for flexible matching: ```yaml theme={null} event: topic: "assets.*" # Matches assets.new, assets.updated, etc. topic: "*.new" # Matches assets.new, vulns.new, etc. topic: "scan.*.complete" # Matches scan.nuclei.complete, scan.httpx.complete topic: "*" # Matches all topics ``` | Pattern | Description | | ------- | ---------------------------------- | | `*` | Matches any sequence of characters | | `?` | Matches any single character | | `[abc]` | Matches any character in the set | ### JavaScript Filters Filters are JavaScript expressions evaluated against each event. All filters must return `true` for the event to trigger the workflow. #### Available Event Fields ```javascript theme={null} event.topic // "assets.new" event.id // "uuid-string" event.name // "subdomain.discovered" event.source // "amass" event.data_type // "subdomain" event.data // Parsed JSON object or raw string event.workspace // "myworkspace" ``` #### Filter Examples ```yaml theme={null} filters: # Match specific source - "event.source == 'nuclei'" # Match severity from parsed data - "event.data.severity == 'critical'" # Match by data type - "event.data_type == 'vulnerability'" # Combine conditions (implicit AND) - "event.source == 'nuclei'" - "event.data.severity == 'high' || event.data.severity == 'critical'" ``` #### Common Filter Patterns ```yaml theme={null} # Only process from specific tools filters: - "event.source == 'subfinder' || event.source == 'amass'" # Filter by severity filters: - "['critical', 'high'].includes(event.data.severity)" # Match URL patterns filters: - "event.data.url && event.data.url.includes('/api/')" # Workspace-specific filters: - "event.workspace == 'production'" ``` ### Filter Functions For more advanced filtering, use `filter_functions` which provides access to utility functions like `contains()`, `starts_with()`, `ends_with()`, `file_exists()`, and more: ```yaml theme={null} triggers: - name: on-api-endpoint on: event event: topic: "assets.new" # Simple filters (basic JS only) filters: - "event.source == 'httpx'" # Filter functions with utility functions available filter_functions: - "contains(event.data.url, '/api/')" - "!ends_with(event.data.url, '.js')" input: type: event_data field: "url" name: target enabled: true ``` #### Available Filter Functions | Function | Description | Example | | -------------------------- | ---------------------------------- | -------------------------------------------------------- | | `contains(str, substr)` | Check if string contains substring | `contains(event.data.url, '/api/')` | | `starts_with(str, prefix)` | Check if string starts with prefix | `starts_with(event.data.severity, 'critical')` | | `ends_with(str, suffix)` | Check if string ends with suffix | `ends_with(event.data.url, '.json')` | | `file_exists(path)` | Check if file exists | `file_exists("{{event.data.output_path}}/results.json")` | | `file_length(path)` | Get file line count | `file_length(event.data.results_file) > 0` | | `is_empty(str)` | Check if string is empty | `!is_empty(event.data.target)` | | `trim(str)` | Remove leading/trailing whitespace | Used in input expressions | #### Combining Filters and Filter Functions You can use both `filters` (basic JS) and `filter_functions` (with utilities) together. All expressions from both must pass: ```yaml theme={null} event: topic: "vulns.discovered" # Basic JS expressions filters: - "event.source == 'nuclei'" # Expressions with utility functions filter_functions: - "contains(event.data.template_id, 'CVE')" - "starts_with(event.data.severity, 'critical') || starts_with(event.data.severity, 'high')" ``` ### Event Envelope Template Variables Event-triggered workflows have access to special template variables containing the full event data: | Variable | Description | | -------------------- | -------------------------------------- | | `{{EventEnvelope}}` | Full JSON-encoded event envelope | | `{{EventTopic}}` | Event topic (e.g., "assets.new") | | `{{EventSource}}` | Event source (e.g., "nuclei", "httpx") | | `{{EventDataType}}` | Data type (e.g., "subdomain", "url") | | `{{EventTimestamp}}` | Event timestamp (RFC3339 format) | | `{{EventData}}` | Raw event data payload | **Example usage:** ```yaml theme={null} name: event-processor kind: module triggers: - name: on-event on: event event: topic: "scan.complete" input: type: event_data field: "target" name: target enabled: true steps: - name: log-event-details type: function functions: - 'log_info("Topic: {{EventTopic}}")' - 'log_info("Source: {{EventSource}}")' - 'log_info("Full envelope: {{EventEnvelope}}")' - name: save-envelope type: bash command: | echo '{{EventEnvelope}}' > {{Output}}/event-envelope.json ``` ### Deduplication Prevent duplicate workflow triggers using time-windowed deduplication: ```yaml theme={null} triggers: - name: on-unique-url on: event event: topic: "crawler.url" dedupe_key: "{{event.data.url}}" # Unique key template dedupe_window: "5m" # Time window input: type: event_data field: "value" name: target ``` Dedupe key supports template variables: * `{{event.source}}-{{event.data.url}}` - Composite key * `{{event.data.hash}}` - Simple field ## Event Topics Reference ### Built-in Topics | Topic | Description | Typical Source | | -------------------- | ---------------------------- | -------------- | | `run.started` | Workflow run began | Engine | | `run.completed` | Workflow run finished | Engine | | `run.failed` | Workflow run failed | Engine | | `step.completed` | Individual step finished | Engine | | `step.failed` | Individual step failed | Engine | | `asset.discovered` | New asset discovered | Recon tools | | `asset.updated` | Asset information updated | Scanners | | `webhook.received` | External webhook received | API | | `schedule.triggered` | Scheduled workflow triggered | Scheduler | ### Custom Topics Define your own topics for domain-specific events: ```javascript theme={null} // Custom discovery events generate_event("{{Workspace}}", "discovery.api-endpoint", "custom-scanner", "endpoint", {...}) // Custom notification events generate_event("{{Workspace}}", "notification.slack", "workflow", "message", {...}) ``` ## Building Event Pipelines ### Chaining Workflows Create pipelines where each workflow triggers the next: ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Subdomain Enum │────▶│ HTTP Probing │────▶│ Vuln Scanning │ │ │ │ │ │ │ │ emit: assets.new│ │ emit: │ │ emit: │ │ (subdomains) │ │ assets.new │ │ vulnerabilities │ │ │ │ (live hosts) │ │ .new │ └─────────────────┘ └─────────────────┘ └─────────────────┘ ``` #### Stage 1: Subdomain Enumeration ```yaml theme={null} name: subdomain-enum kind: module steps: - name: enumerate type: bash command: | subfinder -d {{target}} -o {{Output}}/subdomains.txt amass enum -d {{target}} >> {{Output}}/subdomains.txt sort -u {{Output}}/subdomains.txt -o {{Output}}/subdomains.txt - name: emit-subdomains type: function function: | generate_event_from_file("{{Workspace}}", "assets.new", "recon", "subdomain", "{{Output}}/subdomains.txt") ``` #### Stage 2: HTTP Probing (Triggered by Stage 1) ```yaml theme={null} name: http-probe kind: module triggers: - name: on-subdomain on: event event: topic: "assets.new" filters: - "event.data_type == 'subdomain'" input: type: event_data field: "value" name: target enabled: true params: - name: target required: true steps: - name: probe type: bash command: httpx -u {{target}} -o {{Output}}/probe.json -json - name: emit-if-live type: function pre_condition: 'fileLength("{{Output}}/probe.json") > 0' function: | generate_event("{{Workspace}}", "assets.new", "httpx", "url", readFile("{{Output}}/probe.json")) ``` #### Stage 3: Vulnerability Scanning (Triggered by Stage 2) ```yaml theme={null} name: vuln-scan kind: module triggers: - name: on-live-host on: event event: topic: "assets.new" filters: - "event.source == 'httpx'" - "event.data_type == 'url'" input: type: function function: 'jq("{{event.data}}", ".url")' name: target enabled: true params: - name: target required: true steps: - name: scan type: bash command: nuclei -u {{target}} -o {{Output}}/vulns.json -json - name: emit-findings type: function pre_condition: 'fileLength("{{Output}}/vulns.json") > 0' function: | generate_event_from_file("{{Workspace}}", "vulnerabilities.new", "nuclei", "finding", "{{Output}}/vulns.json") ``` ### Error Handling Pattern Handle failures gracefully with error events: ```yaml theme={null} steps: - name: scan type: bash command: nuclei -u {{target}} on_error: - action: run type: function function: | generate_event("{{Workspace}}", "errors.scan-failed", "nuclei", "error", { target: "{{target}}", step: "scan", error: "Nuclei scan failed" }) ``` ## Real-World Examples ### Asset Discovery Pipeline Complete subdomain discovery to live host probing: ```yaml theme={null} name: asset-discovery-pipeline kind: module description: Discover subdomains and probe for live hosts params: - name: target required: true description: Root domain to enumerate steps: - name: passive-enum type: bash command: | subfinder -d {{target}} -o {{Output}}/subfinder.txt amass enum -passive -d {{target}} -o {{Output}}/amass.txt - name: merge-results type: function functions: - 'exec_cmd("cat {{Output}}/subfinder.txt {{Output}}/amass.txt | sort -u > {{Output}}/all-subs.txt")' - name: probe-live type: bash command: httpx -l {{Output}}/all-subs.txt -o {{Output}}/live-hosts.txt - name: emit-discoveries type: function function: | generate_event_from_file("{{Workspace}}", "assets.new", "pipeline", "live-host", "{{Output}}/live-hosts.txt") log_info("Emitted " + fileLength("{{Output}}/live-hosts.txt") + " live hosts") ``` ### Vulnerability Notification Automatically notify on critical findings: ```yaml theme={null} name: vuln-notifier kind: module triggers: - name: on-critical-vuln on: event event: topic: "vulnerabilities.new" filters: - "event.source == 'nuclei'" - "event.data.severity == 'critical' || event.data.severity == 'high'" input: type: event_data field: "template" name: template_id enabled: true params: - name: template_id required: true steps: - name: notify type: function function: | notify_webhook("ALERT: Critical vulnerability " + "{{template_id}}" + " detected!") notify_telegram("Critical finding: {{template_id}}") ``` ### External Integration: GitHub Webhooks Trigger scans on repository pushes: ```yaml theme={null} name: github-triggered-scan kind: module triggers: - name: on-github-push on: event event: topic: "webhook.received" filters: - "event.source == 'github'" - "event.data.ref == 'refs/heads/main'" input: type: function function: 'jq("{{event.data}}", ".repository.html_url")' name: repo_url enabled: true params: - name: repo_url required: true steps: - name: clone-and-scan type: bash command: | git clone {{repo_url}} /tmp/repo trufflehog filesystem /tmp/repo --json > {{Output}}/secrets.json ``` ## Monitoring Events ### Event Logs API Query event history via the REST API: ```bash theme={null} # List recent events curl "http://localhost:8080/osm/api/event-logs" \ -H "Authorization: Bearer $TOKEN" # Filter by topic curl "http://localhost:8080/osm/api/event-logs?topic=assets.new" \ -H "Authorization: Bearer $TOKEN" # Filter by run curl "http://localhost:8080/osm/api/event-logs?run_id=run-abc123" \ -H "Authorization: Bearer $TOKEN" # Filter by workspace curl "http://localhost:8080/osm/api/event-logs?workspace=production" \ -H "Authorization: Bearer $TOKEN" # Filter by processed status curl "http://localhost:8080/osm/api/event-logs?processed=false" \ -H "Authorization: Bearer $TOKEN" ``` ### CLI Event Logs Query event history using the database CLI: ```bash theme={null} # List recent events (default columns: topic, source, processed, data_type, workspace, data) osmedeus db list --table event_logs # List all available columns osmedeus db list --table event_logs --list-columns # Filter by specific columns osmedeus db list --table event_logs --columns topic,source,data_type,data # Show all columns including hidden ones (id, timestamps) osmedeus db list --table event_logs --all # Filter by topic osmedeus db list --table event_logs --where topic=assets.new # Filter by processed status osmedeus db list --table event_logs --where processed=false # Search events across all columns osmedeus db list --table event_logs --search "nuclei" # Output as JSON for scripting osmedeus db list --table event_logs --json # Pagination osmedeus db list --table event_logs --offset 50 --limit 100 # Interactive TUI mode (default without --table) osmedeus db list ``` ### Queue Metrics The scheduler tracks event processing metrics: | Metric | Description | | -------------------- | -------------------------------- | | `events_enqueued` | Total events successfully queued | | `events_dropped` | Events dropped due to full queue | | `queue_current_size` | Current events waiting | Access via health endpoint: ```bash theme={null} curl http://localhost:8080/osm/api/health ``` ### Event Receiver Status Check the status of event-triggered workflows: ```bash theme={null} # Get event receiver status curl "http://localhost:8080/osm/api/event-receiver/status" \ -H "Authorization: Bearer $TOKEN" # List registered event-triggered workflows curl "http://localhost:8080/osm/api/event-receiver/workflows" \ -H "Authorization: Bearer $TOKEN" ``` ## Bulk Event Processing Process multiple targets discovered via events using the `func eval` bulk processing capabilities. ### Processing Targets from File ```bash theme={null} # Process each target in a file (target variable available in script) osmedeus func eval -e 'log_info("Processing: " + target)' -T targets.txt # With concurrency for parallel processing osmedeus func eval -e 'httpGet("https://" + target)' -T targets.txt -c 10 # With additional parameters osmedeus func eval -e 'log_info(target + " in " + workspace)' -T targets.txt --params workspace=production ``` ### Using Function Files Store reusable processing logic in files: ```javascript theme={null} // check-host.js var result = httpGet("https://" + target); if (result.status == 200) { generate_event("myworkspace", "assets.live", "check", "url", target); log_info("Live: " + target); } ``` ```bash theme={null} # Execute function file against targets osmedeus func eval --function-file check-host.js -T discovered-hosts.txt -c 5 ``` ### Function Call Syntax Multiple ways to invoke functions: ```bash theme={null} # Expression with -e flag osmedeus func eval -e 'log_info("hello")' # Positional argument osmedeus func eval 'log_info("hello")' # Function name with arguments osmedeus func eval log_info "hello world" # Using -f flag for function name osmedeus func eval -f log_info "hello world" # Read from stdin echo 'log_info("hello")' | osmedeus func eval --stdin ``` ### Integration with Event Pipeline Combine event log queries with bulk function evaluation: ```bash theme={null} # 1. Export discovered assets to file osmedeus db list --table event_logs --where data_type=subdomain --json | \ jq -r '.[].data' > /tmp/subdomains.txt # 2. Process with bulk function evaluation osmedeus func eval -e 'httpGet("https://" + target)' -T /tmp/subdomains.txt -c 10 # 3. Generate events for results osmedeus func eval --function-file process-subdomain.js -T /tmp/subdomains.txt -c 5 ``` ### Testing Event Functions Test event generation before deploying workflows: ```bash theme={null} # Test single event generation osmedeus func eval 'generate_event("test-workspace", "test.topic", "cli", "test", "hello")' # Test with target variable osmedeus func eval -e 'generate_event("ws", "assets.new", "test", "url", target)' -t example.com # List available event functions osmedeus func list event ``` ## Best Practices 1. **Use specific topics** - Prefer `assets.subdomain` over generic `assets.new` for precise filtering 2. **Filter early** - Apply filters to reduce processing overhead and prevent unnecessary workflow triggers 3. **Handle backpressure gracefully** - Design workflows to tolerate dropped events during high load 4. **Log event errors** - Use `on_error` handlers to track and emit failure events for debugging 5. **Test triggers disabled first** - Set `enabled: false` initially, validate filters, then enable 6. **Use idempotent handlers** - Workflows may receive duplicate events; design steps to handle this 7. **Batch file emissions** - Use `generate_event_from_file` for bulk discoveries instead of individual events 8. **Use deduplication** - Configure `dedupe_key` and `dedupe_window` to prevent duplicate processing 9. **Include workspace** - Always pass the workspace parameter to maintain proper event isolation ## Troubleshooting ### Events Not Triggering 1. **Check topic match**: Event topic must exactly match trigger configuration ```bash theme={null} # Verify event topics in database osmedeus db list --table event_logs --columns topic,source,data_type --limit 10 ``` 2. **Verify trigger is enabled**: `enabled: true` in workflow YAML 3. **Test filter expressions**: Simplify filters to isolate issues ```yaml theme={null} # Start with no filters filters: [] # Then add back one at a time filters: - "event.source == 'nuclei'" ``` 4. **Check scheduler is running**: Events only process when server is active 5. **Check event receiver status**: ```bash theme={null} curl "http://localhost:8080/osm/api/event-receiver/status" ``` ### Events Dropped 1. **Check queue metrics**: High `events_dropped` indicates overload 2. **Reduce event volume**: Batch discoveries, filter at source 3. **Increase queue size**: Configure via scheduler settings if needed 4. **Scale with workers**: Distribute processing across workers ### Filter Not Matching 1. **Verify event data structure**: Check actual event payload ```bash theme={null} osmedeus db list --table event_logs --where topic=assets.new --json --limit 1 ``` 2. **Test JS expression**: Filters use JavaScript syntax ```javascript theme={null} // Correct event.data.severity == 'critical' // Wrong (using Go syntax) event.data.severity = "critical" ``` 3. **Check data types**: String vs number comparisons ```javascript theme={null} // If status_code is number event.data.status_code == 200 // If status_code is string event.data.status_code == '200' ``` ### Testing Event Functions Use the CLI to test event functions interactively: ```bash theme={null} # Test generate_event osmedeus func eval 'generate_event("test", "test.topic", "cli", "test", "data")' # Verify event was created osmedeus db list --table event_logs --where topic=test.topic --limit 1 ``` # Using Osmedeus as a Library Source: https://docs.osmedeus.org/advanced/library Embed Osmedeus workflow engine in your Go applications Osmedeus can be used as a Go library to embed workflow execution capabilities in your own applications. ## Installation ```bash theme={null} go get github.com/j3ssie/osmedeus/v5 ``` ## Quick Start ```go theme={null} package main import ( "context" "fmt" "log" "github.com/j3ssie/osmedeus/v5/internal/config" "github.com/j3ssie/osmedeus/v5/internal/executor" "github.com/j3ssie/osmedeus/v5/internal/parser" ) func main() { ctx := context.Background() // 1. Load configuration cfg, err := config.NewConfig("") if err != nil { log.Fatal(err) } // 2. Load workflow loader := parser.NewLoader(cfg.WorkflowsPath) workflow, err := loader.LoadWorkflow("my-module") if err != nil { log.Fatal(err) } // 3. Create and configure executor exec := executor.NewExecutor() exec.SetVerbose(true) exec.SetLoader(loader) // 4. Execute workflow result, err := exec.ExecuteModule(ctx, workflow, map[string]string{ "target": "example.com", "tactic": "default", }, cfg) if err != nil { log.Fatal(err) } // 5. Check results fmt.Printf("Status: %s\n", result.Status) for _, step := range result.Steps { fmt.Printf(" - %s: %s\n", step.StepName, step.Status) } } ``` ## Core Components ### Configuration Load Osmedeus configuration from the default location (`~/osmedeus-base/osm-settings.yaml`): ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/config" // Load from default path cfg, err := config.NewConfig("") // Load from custom path cfg, err := config.NewConfig("/path/to/osm-settings.yaml") // Access configuration fmt.Println("Workflows:", cfg.WorkflowsPath) fmt.Println("Binaries:", cfg.BinariesPath) fmt.Println("Data:", cfg.DataPath) ``` ### Workflow Parser Parse and validate workflow YAML files: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/parser" // Create parser p := parser.NewParser() // Parse workflow from file workflow, err := p.Parse("path/to/workflow.yaml") // Parse workflow from bytes workflow, err := p.ParseContent([]byte(yamlContent)) // Validate workflow if err := p.Validate(workflow); err != nil { log.Fatal("Invalid workflow:", err) } ``` ### Workflow Loader Load workflows by name with caching: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/parser" // Create loader loader := parser.NewLoader("/path/to/workflows") // Load by name (searches modules/ and flows/ directories) workflow, err := loader.LoadWorkflow("subdomain-enum") // Load by path workflow, err := loader.LoadWorkflow("/path/to/custom.yaml") // Load all workflows workflows, err := loader.LoadAllWorkflows() // Reload from disk (clears cache) err := loader.ReloadWorkflows() // Get cached workflow workflow, found := loader.GetWorkflow("name") ``` ### Executor Execute workflows programmatically: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/executor" // Create executor exec := executor.NewExecutor() // Configure execution options exec.SetDryRun(true) // Show commands without executing exec.SetVerbose(true) // Show step output exec.SetSilent(true) // Hide step output exec.SetSpinner(true) // Show spinner animation exec.SetServerMode(true) // Enable file logging exec.SetLoader(loader) // Required for flow execution // Execute module workflow result, err := exec.ExecuteModule(ctx, workflow, params, cfg) // Execute flow workflow result, err := exec.ExecuteFlow(ctx, flowWorkflow, params, cfg) ``` #### Execution Parameters ```go theme={null} params := map[string]string{ "target": "example.com", // Target to scan "tactic": "default", // aggressive, default, gently "threads_hold": "10", // Override thread count "workspace_prefix": "prefix", // Workspace folder suffix "workspaces_folder": "/path", // Override workspaces directory "exclude_modules": "mod1,mod2", // Modules to skip (flows) "heuristics_check": "basic", // none, basic, advanced } ``` #### Handling Results ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/core" result, err := exec.ExecuteModule(ctx, workflow, params, cfg) if err != nil { log.Fatal(err) } // Check overall status switch result.Status { case core.RunStatusCompleted: fmt.Println("Workflow completed successfully") case core.RunStatusFailed: fmt.Println("Workflow failed:", result.Error) case core.RunStatusCancelled: fmt.Println("Workflow was cancelled") } // Iterate step results for _, step := range result.Steps { fmt.Printf("Step: %s, Status: %s\n", step.StepName, step.Status) if step.Output != "" { fmt.Printf(" Output: %s\n", step.Output) } } // Access exports for key, value := range result.Exports { fmt.Printf("Export: %s = %v\n", key, value) } ``` ## Template Engine Render template strings with variable interpolation: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/template" // Create engine engine := template.NewEngine() // Render single template result, err := engine.Render("Hello {{Name}}", map[string]any{ "Name": "World", }) // result = "Hello World" // Render multiple templates results, err := engine.RenderMap(map[string]string{ "greeting": "Hello {{Name}}", "command": "scan {{Target}}", }, variables) // Render slice results, err := engine.RenderSlice([]string{ "{{var1}}", "{{var2}}", }, variables) ``` ### Secondary Variables (for loops) Use `[[variable]]` syntax for loop variables to avoid conflicts: ```go theme={null} // For foreach contexts result, err := engine.RenderSecondary( "Processing [[item]] with ID [[_id_]]", map[string]any{"item": "value", "_id_": 5}, ) // Check if template uses secondary delimiters if engine.HasSecondaryVariable(template) { result, err = engine.RenderSecondary(template, vars) } ``` ### Generator Functions Execute generator functions in templates: ```go theme={null} // Generate UUID uuid, err := engine.ExecuteGenerator("uuid()") // Get current date date, err := engine.ExecuteGenerator("currentDate(\"2006-01-02\")") // Get environment variable value, err := engine.ExecuteGenerator("getEnvVar(\"HOME\", \"/tmp\")") ``` ## Function Registry Execute utility functions programmatically: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/functions" // Create registry registry := functions.NewRegistry() // Execute expression result, err := registry.Execute("trim(' hello ')", variables) // Evaluate condition (returns bool) ok, err := registry.EvaluateCondition( "fileExists('{{Output}}/results.txt')", variables, ) // Evaluate exports exports, err := registry.EvaluateExports(map[string]string{ "line_count": "fileLength('{{Output}}/data.txt')", "has_data": "fileExists('{{Output}}/data.txt')", }, variables) ``` ### Available Functions | Category | Functions | | -------- | --------------------------------------------------------------------------------- | | File | `fileExists`, `fileLength`, `readFile`, `writeFile`, `appendFile`, `createFolder` | | String | `trim`, `split`, `contains`, `indexOf`, `toUpper`, `toLower`, `replace` | | JSON | `jq`, `jqFromFile`, `jsonl2csv`, `filterJsonl` | | Database | `db_select`, `db_insert`, `db_update`, `db_delete` | | Logging | `log_info`, `log_warn`, `log_error` | ## Execution Context Manage workflow execution state: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/core" // Create context ctx := core.NewExecutionContext( "workflow-name", core.KindModule, "run-123", "example.com", ) // Set variables (thread-safe) ctx.SetVariable("key", "value") ctx.SetParam("param_name", value) ctx.SetExport("export_name", value) // Get variables val, ok := ctx.GetVariable("key") allVars := ctx.GetVariables() // For template rendering ``` ## Database Access Connect to the Osmedeus database: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/database" // Connect based on config db, err := database.Connect(cfg) // Run migrations err = database.Migrate(context.Background()) // Query using Bun ORM var workspaces []database.Workspace err := db.NewSelect(). Model(&workspaces). Where("name = ?", "example.com"). Scan(context.Background()) ``` ### Database Models ```go theme={null} // Workspace database.Workspace{ Name string Path string TotalAssets int } // Asset database.Asset{ ID string WorkspaceID string Type string // domain, subdomain, url Value string Source string } // Run database.Run{ ID string WorkspaceID string Status string StartedAt time.Time CompletedAt time.Time } ``` ## Core Types ### Workflow Types ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/core" // Workflow kinds core.KindModule // Single execution unit core.KindFlow // Orchestrates modules core.KindFragment // Reusable step collections // Step types core.StepTypeBash // Shell commands core.StepTypeFunction // Utility functions core.StepTypeForeach // Loop iteration core.StepTypeParallelSteps // Concurrent execution core.StepTypeRemoteBash // Remote execution core.StepTypeHTTP // HTTP requests core.StepTypeLLM // AI-powered steps // Runner types core.RunnerTypeHost // Local execution core.RunnerTypeDocker // Container execution core.RunnerTypeSSH // Remote SSH execution // Run status core.RunStatusPending core.RunStatusRunning core.RunStatusCompleted core.RunStatusFailed core.RunStatusCancelled ``` ### Step Result ```go theme={null} type StepResult struct { StepName string Status StepStatus Output string Error error Exports map[string]interface{} Duration time.Duration LogFile string } ``` ### Workflow Result ```go theme={null} type WorkflowResult struct { WorkflowName string WorkflowKind WorkflowKind RunID string Target string Status RunStatus Steps []*StepResult Exports map[string]interface{} Error error } ``` ## Creating Custom Workflows Build workflows programmatically: ```go theme={null} workflow := &core.Workflow{ Kind: core.KindModule, Name: "custom-scan", Description: "Custom scanning workflow", Params: []core.Param{ {Name: "target", Required: true}, }, Steps: []core.Step{ { Name: "enumerate", Type: core.StepTypeBash, Command: "subfinder -d {{target}} -o {{Output}}/subs.txt", Exports: map[string]string{ "sub_count": "fileLength('{{Output}}/subs.txt')", }, }, { Name: "probe", Type: core.StepTypeBash, Command: "httpx -l {{Output}}/subs.txt -o {{Output}}/live.txt", PreCondition: "{{sub_count}} > 0", }, }, } // Validate before execution p := parser.NewParser() if err := p.Validate(workflow); err != nil { log.Fatal(err) } // Execute result, err := exec.ExecuteModule(ctx, workflow, params, cfg) ``` ## Workflow Linting Validate workflows with the linter: ```go theme={null} import "github.com/j3ssie/osmedeus/v5/internal/linter" // Create linter l := linter.NewLinter() // Lint workflow issues := l.Lint(workflow) for _, issue := range issues { fmt.Printf("[%s] %s: %s at %s\n", issue.Severity, // info, warning, error issue.Rule, issue.Message, issue.Location, ) } // Filter by severity warnings := l.LintWithSeverity(workflow, linter.SeverityWarning) ``` ## Best Practices 1. **Always validate workflows** before execution 2. **Use context cancellation** for long-running workflows 3. **Handle errors** from each step appropriately 4. **Use the loader** for flow execution (required for module resolution) 5. **Set appropriate timeouts** for network operations 6. **Use dry-run mode** for testing workflow logic ## Complete Example ```go theme={null} package main import ( "context" "fmt" "log" "os" "os/signal" "syscall" "github.com/j3ssie/osmedeus/v5/internal/config" "github.com/j3ssie/osmedeus/v5/internal/core" "github.com/j3ssie/osmedeus/v5/internal/database" "github.com/j3ssie/osmedeus/v5/internal/executor" "github.com/j3ssie/osmedeus/v5/internal/parser" ) func main() { // Setup context with cancellation ctx, cancel := context.WithCancel(context.Background()) defer cancel() // Handle interrupt sigCh := make(chan os.Signal, 1) signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) go func() { <-sigCh fmt.Println("\nCancelling workflow...") cancel() }() // Load configuration cfg, err := config.NewConfig("") if err != nil { log.Fatal("Failed to load config:", err) } // Connect to database _, err = database.Connect(cfg) if err != nil { log.Fatal("Failed to connect to database:", err) } database.Migrate(ctx) // Create workflow loader loader := parser.NewLoader(cfg.WorkflowsPath) // Load workflow workflow, err := loader.LoadWorkflow("subdomain-enum") if err != nil { log.Fatal("Failed to load workflow:", err) } // Validate workflow p := parser.NewParser() if err := p.Validate(workflow); err != nil { log.Fatal("Invalid workflow:", err) } // Create and configure executor exec := executor.NewExecutor() exec.SetVerbose(true) exec.SetLoader(loader) // Execute workflow result, err := exec.ExecuteModule(ctx, workflow, map[string]string{ "target": "example.com", "tactic": "default", }, cfg) if err != nil { log.Fatal("Execution failed:", err) } // Report results fmt.Printf("\n=== Workflow Complete ===\n") fmt.Printf("Status: %s\n", result.Status) fmt.Printf("Steps executed: %d\n", len(result.Steps)) for _, step := range result.Steps { status := "✓" if step.Status != core.StepStatusSuccess { status = "✗" } fmt.Printf(" %s %s (%s)\n", status, step.StepName, step.Duration) } if len(result.Exports) > 0 { fmt.Println("\nExports:") for k, v := range result.Exports { fmt.Printf(" %s: %v\n", k, v) } } } ``` ## Next Steps * [Extending Osmedeus](extending-osmedeus) - Add custom step types and runners * [Development](development) - Set up development environment * [REST API](../reference/rest-api) - HTTP API for workflow execution # LLM & Agents Integration Source: https://docs.osmedeus.org/advanced/llm AI-powered workflow steps using Large Language Models. ## Overview Osmedeus provides three step types for LLM integration: * **`llm`** — Single-shot LLM calls: chat completions, tool calling, embeddings, multimodal content, structured outputs * **`agent`** — Agentic execution loop: iterative tool calling, sub-agents, memory management, planning stages, multi-goal execution * **`agent-acp`** — External AI agent execution via the [Agent Communication Protocol (ACP)](https://github.com/anthropics/agent-communication-protocol): delegates to real agent binaries (Claude Code, Codex, OpenCode, Gemini) ## Configuration ### Settings In `osm-settings.yaml`, configure one or more providers under `llm_providers`. Providers are rotated automatically across requests: ```yaml theme={null} llm: llm_providers: - provider: openai base_url: "https://api.openai.com/v1" auth_token: "sk-..." model: gpt-4 - provider: anthropic base_url: "https://api.anthropic.com/v1" auth_token: "sk-ant-..." model: claude-3-opus max_tokens: 4096 temperature: 0.7 stream: false ``` ### Environment Variables Environment variables override settings for the default provider: ```bash theme={null} export OSM_LLM_BASE_URL=https://api.openai.com/v1 export OSM_LLM_AUTH_TOKEN=sk-... export OSM_LLM_MODEL=gpt-4 ``` ## Chat Completion ### Basic Usage ```yaml theme={null} - name: analyze-results type: llm messages: - role: system content: You are a security analyst. Analyze findings concisely. - role: user content: | Analyze these vulnerabilities: {{readFile("{{Output}}/vulns.txt")}} exports: analysis: "{{analyze_results_content}}" ``` Export variables are based on the **sanitized step name** (hyphens replaced with underscores). A step named `analyze-results` produces exports `analyze_results_llm_resp` (full response object) and `analyze_results_content` (text content only). ### Message Roles | Role | Description | | ----------- | -------------------- | | `system` | System instructions | | `user` | User input | | `assistant` | Previous AI response | | `tool` | Tool call result | ### Multi-turn Conversation ```yaml theme={null} - name: chat type: llm messages: - role: system content: You are a helpful security assistant. - role: user content: What is SQL injection? - role: assistant content: SQL injection is a code injection technique... - role: user content: How do I prevent it in Python? ``` ## Tool Calling ### Define Tools ```yaml theme={null} - name: intelligent-scan type: llm messages: - role: system content: You are a security scanner. Use tools to analyze targets. - role: user content: Analyze {{target}} for security issues. tools: - type: function function: name: port_scan description: Scan ports on a target parameters: type: object properties: target: type: string description: Target IP or hostname ports: type: string description: Port range (e.g., "1-1000") required: ["target"] - type: function function: name: vulnerability_scan description: Run vulnerability scan parameters: type: object properties: target: type: string templates: type: string enum: ["cves", "misconfigurations", "exposures"] required: ["target"] ``` ### Handle Tool Calls Tool calls are exported within the `_llm_resp` object: ```yaml theme={null} - name: ai-scan type: llm messages: - role: user content: Scan {{target}} tools: - type: function function: name: scan parameters: { ... } exports: full_response: "{{ai_scan_llm_resp}}" - name: execute-tool type: function pre_condition: '{{full_response}} != ""' function: | // Parse and execute tool calls executeToolCalls("{{full_response}}") ``` ## Embeddings ### Generate Embeddings ```yaml theme={null} - name: embed-findings type: llm is_embedding: true embedding_input: - "SQL injection in login form" - "Cross-site scripting in search" - "Insecure direct object reference" exports: embeddings: "{{embed_findings_llm_resp}}" ``` ### Use with Files ```yaml theme={null} - name: embed-vulns type: llm is_embedding: true embedding_input: "{{readLines('{{Output}}/vulns.txt')}}" exports: vuln_embeddings: "{{embed_vulns_llm_resp}}" ``` ## Structured Output ### JSON Schema ```yaml theme={null} - name: extract-findings type: llm messages: - role: user content: | Extract vulnerabilities from this report: {{readFile("{{Output}}/scan-report.txt")}} response_format: type: json_schema json_schema: name: vulnerabilities schema: type: object properties: findings: type: array items: type: object properties: title: type: string severity: type: string enum: ["critical", "high", "medium", "low"] description: type: string required: ["findings"] exports: structured_findings: "{{extract_findings_content}}" ``` ## Configuration Override ### Per-Step Config ```yaml theme={null} - name: local-analysis type: llm llm_config: provider: ollama model: llama2 max_tokens: 2048 temperature: 0.5 stream: true messages: - role: user content: Analyze {{target}} ``` ### Extra Parameters ```yaml theme={null} - name: creative-analysis type: llm messages: - role: user content: Write a security assessment for {{target}} extra_llm_parameters: temperature: 0.9 top_p: 0.95 frequency_penalty: 0.5 ``` ## Multimodal Content ### Image Analysis ```yaml theme={null} - name: analyze-screenshot type: llm messages: - role: user content: - type: text text: Analyze this screenshot for security issues. - type: image_url image_url: url: "file://{{Output}}/screenshot.png" ``` ## Streaming Both `llm` and `agent` steps support streaming output via the `stream` field: ```yaml theme={null} - name: stream-analysis type: llm stream: true messages: - role: user content: Analyze {{target}} in detail. ``` The `stream` field overrides both `llm_config.stream` and the global config setting. *** ## Agent Step Type The `agent` step type provides an **agentic LLM execution loop** — the LLM iteratively calls tools, processes results, and reasons until completion. This is fundamentally different from the single-shot `llm` step. ### Basic Agent ```yaml theme={null} - name: recon-agent type: agent query: "Enumerate subdomains of {{Target}} and identify interesting services." system_prompt: "You are an expert security reconnaissance agent." max_iterations: 15 agent_tools: - preset: bash - preset: read_file - preset: save_content exports: findings: "{{agent_content}}" ``` | Field | Type | Required | Description | | ---------------- | --------------- | -------- | -------------------------------------------------- | | `query` | string | Yes\* | The task prompt for the agent | | `queries` | string\[] | Yes\* | Multiple goals executed sequentially | | `system_prompt` | string | No | System prompt for the agent | | `max_iterations` | int | Yes | Maximum tool-calling loop iterations (must be > 0) | | `agent_tools` | AgentToolDef\[] | No | Tools available to the agent | Either `query` (single goal) or `queries` (multi-goal) is required, not both. ### Preset Tools Preset tools reference built-in osmedeus functions with auto-generated schemas: ```yaml theme={null} agent_tools: - preset: bash - preset: read_file - preset: grep_regex ``` | Preset | Description | | ------------------ | ------------------------------------------------------------- | | `bash` | Execute a shell command and return its output | | `read_file` | Read the contents of a file | | `read_lines` | Read a file and return its contents as an array of lines | | `file_exists` | Check if a file exists at the given path | | `file_length` | Count the number of non-empty lines in a file | | `append_file` | Append content from source file to destination file | | `save_content` | Write string content to a file (overwrites if exists) | | `glob` | Find files matching a glob pattern | | `grep_string` | Search a file for lines containing a string | | `grep_regex` | Search a file for lines matching a regex pattern | | `http_get` | Make an HTTP GET request and return the response | | `http_request` | Make an HTTP request with specified method, headers, and body | | `jq` | Query JSON data using jq expression syntax | | `exec_python` | Run inline Python code and return stdout | | `exec_python_file` | Run a Python file and return stdout | | `exec_ts` | Run inline TypeScript code via bun and return stdout | | `exec_ts_file` | Run a TypeScript file via bun and return stdout | | `run_module` | Run an osmedeus module as a subprocess | | `run_flow` | Run an osmedeus flow as a subprocess | ### Custom Tools Define custom tools with explicit schemas and JavaScript handlers: ```yaml theme={null} agent_tools: - preset: bash - name: check_port description: "Check if a port is open on a host" parameters: type: object properties: host: type: string description: "Target hostname or IP" port: type: integer description: "Port number to check" required: ["host", "port"] handler: | exec("nc -zv -w3 " + args.host + " " + args.port) ``` The `handler` is a JavaScript expression. The parsed tool call arguments are available as the `args` object. ### Multi-Goal Execution Use `queries` to run the agent through multiple goals sequentially. Each goal is executed in order, and all results are collected: ```yaml theme={null} - name: full-recon type: agent queries: - "Discover all subdomains of {{Target}}" - "Identify web services running on discovered subdomains" - "Check for common misconfigurations on each service" system_prompt: "You are a thorough security auditor." max_iterations: 20 agent_tools: - preset: bash - preset: read_file - preset: save_content exports: all_results: "{{agent_goal_results}}" final_output: "{{agent_content}}" ``` The `agent_goal_results` export contains results from all goals as a JSON array. ### Planning Stage Add a planning phase before the main execution loop. The agent first generates a plan, then executes it: ```yaml theme={null} - name: planned-scan type: agent query: "Perform a comprehensive security assessment of {{Target}}" plan_prompt: | Create a step-by-step plan for assessing {{Target}}. Consider: subdomain enumeration, service detection, vulnerability scanning. plan_max_tokens: 1000 max_iterations: 25 agent_tools: - preset: bash - preset: read_file - preset: save_content exports: plan: "{{agent_plan}}" results: "{{agent_content}}" ``` | Field | Type | Description | | ----------------- | ------ | ------------------------------------------------------------------------- | | `plan_prompt` | string | Prompt for the planning phase (triggers plan generation before main loop) | | `plan_max_tokens` | int | Max tokens for the plan response | ### Memory Management Control conversation context size for long-running agents: ```yaml theme={null} - name: long-running-agent type: agent query: "Perform deep reconnaissance on {{Target}}" max_iterations: 50 memory: max_messages: 30 summarize_on_truncate: true persist_path: "{{Output}}/agent/conversation.json" resume_path: "{{Output}}/agent/conversation.json" agent_tools: - preset: bash - preset: read_file - preset: save_content ``` | Field | Type | Default | Description | | ----------------------- | ------ | ------------- | ------------------------------------------------------------------------- | | `max_messages` | int | 0 (unlimited) | Sliding window size; oldest non-system messages are dropped when exceeded | | `summarize_on_truncate` | bool | false | Use LLM to summarize dropped messages instead of silently discarding them | | `persist_path` | string | — | Save conversation JSON after completion | | `resume_path` | string | — | Load a prior conversation on start (enables continuation across runs) | ### Model Preferences Specify preferred models tried in order. Falls back to the default provider config if none are available: ```yaml theme={null} - name: smart-agent type: agent query: "Analyze complex target architecture for {{Target}}" max_iterations: 10 models: - claude-3-opus - gpt-4 - claude-3-sonnet agent_tools: - preset: bash ``` ### Structured Output (Agent) Enforce a JSON schema on the agent's final output using `output_schema`: ```yaml theme={null} - name: structured-agent type: agent query: "Find all open ports and services on {{Target}}" max_iterations: 15 output_schema: '{"type":"object","properties":{"ports":{"type":"array","items":{"type":"object","properties":{"port":{"type":"integer"},"service":{"type":"string"},"version":{"type":"string"}}}},"summary":{"type":"string"}},"required":["ports","summary"]}' agent_tools: - preset: bash - preset: save_content exports: structured_results: "{{agent_content}}" ``` The schema is enforced on the final iteration via the OpenAI `response_format` parameter. ### Sub-Agents Define inline sub-agents that the parent agent can spawn via the auto-generated `spawn_agent` tool: ```yaml theme={null} - name: coordinator type: agent query: "Assess {{Target}} using specialized sub-agents for each phase." system_prompt: "You are a coordinator agent. Delegate tasks to specialized sub-agents." max_iterations: 10 max_agent_depth: 3 agent_tools: - preset: read_file - preset: save_content sub_agents: - name: subdomain-scanner description: "Discovers subdomains for a target domain" system_prompt: "You are a subdomain enumeration specialist." max_iterations: 10 agent_tools: - preset: bash - preset: save_content - name: vuln-checker description: "Checks for vulnerabilities on discovered services" system_prompt: "You are a vulnerability assessment specialist." max_iterations: 10 agent_tools: - preset: bash - preset: read_file output_schema: '{"type":"object","properties":{"vulnerabilities":{"type":"array"}}}' exports: assessment: "{{agent_content}}" ``` When `sub_agents` is defined, a `spawn_agent` tool is automatically added with parameters: * `agent` — Name of the sub-agent to spawn (from the defined list) * `query` — The task to delegate Sub-agents support recursive nesting (sub-agents can define their own `sub_agents`). Use `max_agent_depth` to control nesting depth (default: 3). ### Stop Condition A JavaScript expression evaluated after each iteration. If it returns `true`, the agent stops: ```yaml theme={null} - name: targeted-scan type: agent query: "Find the admin panel for {{Target}}" max_iterations: 20 stop_condition: 'agent_content.includes("admin") && iteration > 3' agent_tools: - preset: bash - preset: read_file ``` Available variables in the expression: `agent_content` (current response text), `iteration` (current iteration number). ### Tool Tracing Hooks JavaScript expressions executed before and after each tool call for logging or debugging: ```yaml theme={null} - name: traced-agent type: agent query: "Scan {{Target}}" max_iterations: 10 on_tool_start: 'log_info("Calling tool: " + tool_name + " with: " + tool_args)' on_tool_end: 'log_info("Tool " + tool_name + " returned: " + tool_result.substring(0, 200))' agent_tools: - preset: bash - preset: read_file ``` | Hook | Available Variables | | --------------- | --------------------------------------- | | `on_tool_start` | `tool_name`, `tool_args` | | `on_tool_end` | `tool_name`, `tool_args`, `tool_result` | ### Parallel Tool Calls By default, agents allow the LLM to make multiple tool calls in parallel. Disable this for sequential execution: ```yaml theme={null} - name: sequential-agent type: agent query: "Carefully test {{Target}} one step at a time" max_iterations: 10 parallel_tool_calls: false agent_tools: - preset: bash ``` ### Agent Exports All exports available from agent steps: | Export | Type | Description | | ------------------------- | ------ | ----------------------------------------------- | | `agent_content` | string | Final text response from the agent | | `agent_history` | JSON | Full conversation history | | `agent_iterations` | int | Number of iterations executed | | `agent_total_tokens` | int | Total tokens consumed | | `agent_prompt_tokens` | int | Prompt tokens consumed | | `agent_completion_tokens` | int | Completion tokens consumed | | `agent_tool_results` | JSON | All tool call results | | `agent_plan` | string | Plan content (when `plan_prompt` is used) | | `agent_goal_results` | JSON | Results from each goal (when `queries` is used) | ```yaml theme={null} exports: report: "{{agent_content}}" history: "{{agent_history}}" stats: "{{agent_iterations}}" plan: "{{agent_plan}}" ``` *** ## Agent-ACP Step Type The `agent-acp` step type spawns an **external AI coding agent as a subprocess** and communicates via the [Agent Communication Protocol (ACP)](https://github.com/anthropics/agent-communication-protocol). Unlike the `agent` step type (which uses Osmedeus's internal LLM loop), `agent-acp` delegates to real agent binaries like Claude Code, Codex, OpenCode, or Gemini. ### Basic Agent-ACP ```yaml theme={null} - name: analyze-target type: agent-acp agent: claude-code messages: - role: user content: "Analyze the scan results in {{Output}} and create a summary report." exports: analysis: "{{acp_output}}" ``` ### Built-in Agents Four agents are available out of the box: | Agent Name | Command | Description | | ------------- | ----------------------------------------------- | ----------------------------- | | `claude-code` | `npx -y @zed-industries/claude-code-acp@latest` | Anthropic's Claude Code agent | | `codex` | `npx -y @zed-industries/codex-acp` | OpenAI Codex agent | | `opencode` | `opencode acp` | OpenCode agent | | `gemini` | `gemini --experimental-acp` | Google Gemini agent | List available agents from the CLI: ```bash theme={null} osmedeus agent --list ``` ### Configuration Fields | Field | Type | Required | Description | | --------------- | --------- | -------- | ----------------------------------------------- | | `agent` | string | Yes\* | Built-in agent name (see table above) | | `messages` | array | Yes | Conversation messages with `role` and `content` | | `cwd` | string | No | Working directory for the ACP session | | `allowed_paths` | string\[] | No | Restrict file reads to these directories | | `acp_config` | object | No | Custom ACP configuration (see below) | `agent` is required unless `acp_config.command` is provided to specify a custom agent binary. ### ACP Config Override the built-in agent or customize execution: ```yaml theme={null} acp_config: command: /path/to/custom-agent # Custom agent binary (overrides built-in) args: # Custom command arguments - --flag1 - value1 env: # Environment variables for the agent process CUSTOM_VAR: "value" TARGET_DOMAIN: "{{Target}}" write_enabled: true # Allow agent to write files (default: false) ``` | Field | Type | Default | Description | | --------------- | --------- | ------- | ---------------------------------------------------------- | | `command` | string | — | Custom agent command (overrides built-in registry) | | `args` | string\[] | — | Custom command arguments | | `env` | map | — | Extra environment variables (values are template-rendered) | | `write_enabled` | bool | `false` | Allow the agent to write files | ### Full Configuration Example ```yaml theme={null} - name: security-analysis type: agent-acp agent: claude-code cwd: "{{Output}}" allowed_paths: - "{{Output}}" - "/tmp" acp_config: env: TARGET_DOMAIN: "{{Target}}" write_enabled: true messages: - role: system content: "You are a security analyst. Analyze scan results and produce actionable findings." - role: user content: | Review the scan results in {{Output}} for {{Target}}. Create a prioritized summary of findings. exports: analysis: "{{acp_output}}" agent_name: "{{acp_agent}}" ``` ### Custom Agent Use `acp_config.command` to run any ACP-compatible agent binary: ```yaml theme={null} - name: custom-agent type: agent-acp acp_config: command: /usr/local/bin/my-agent args: - --mode - security env: API_KEY: "{{env_api_key}}" write_enabled: true messages: - role: user content: "Perform analysis on {{Target}}" exports: result: "{{acp_output}}" ``` ### Agent-ACP Exports | Export | Type | Description | | ------------ | ------ | --------------------------------------------- | | `acp_output` | string | Collected stdout from the agent (main output) | | `acp_stderr` | string | Collected stderr from the agent process | | `acp_agent` | string | Name/command of the agent that ran | ```yaml theme={null} exports: result: "{{acp_output}}" errors: "{{acp_stderr}}" agent: "{{acp_agent}}" ``` ### Agent CLI Command Run an ACP agent interactively from the terminal: ```bash theme={null} # Run with claude-code (default) osmedeus agent "Summarize the code in this directory" # Use a specific agent osmedeus agent --agent codex "Explain what this project does" # Set working directory and timeout osmedeus agent --cwd /path/to/project --timeout 1h "Review the code" # Read from stdin echo "Analyze this codebase" | osmedeus agent --stdin cat prompt.txt | osmedeus agent - # List available agents osmedeus agent --list ``` | Flag | Type | Default | Description | | ----------- | ------ | ------------- | --------------------------------------------- | | `--agent` | string | `claude-code` | Agent name to use | | `--cwd` | string | current dir | Working directory for the agent | | `--stdin` | bool | `false` | Read message from stdin | | `--timeout` | string | `30m` | Timeout duration (e.g., `30m`, `1h`, `2h30m`) | | `--list` | bool | `false` | List available agents and exit | ### run\_agent Utility Function Use `run_agent` in function steps to invoke an ACP agent programmatically: ```yaml theme={null} - name: quick-analysis type: function function: 'run_agent("Analyze {{Target}} for security issues", "claude-code")' exports: result: "{{_result}}" # With default agent (claude-code) - name: summarize type: function function: 'run_agent("Summarize the findings in {{Output}}/results.txt")' exports: summary: "{{_result}}" ``` *** ## API Endpoints ### LLM API OpenAI-compatible API: ```bash theme={null} # Chat completion curl -X POST http://localhost:8002/osm/api/llm/v1/chat/completions \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4", "messages": [ {"role": "user", "content": "Analyze this vulnerability: ..."} ] }' # Embeddings curl -X POST http://localhost:8002/osm/api/llm/v1/embeddings \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "model": "text-embedding-ada-002", "input": ["text to embed"] }' ``` ### Agent ACP API OpenAI-compatible endpoint that spawns a local ACP agent subprocess: ```bash theme={null} curl -X POST http://localhost:8002/osm/api/agent/chat/completions \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "model": "claude-code", "messages": [ {"role": "system", "content": "You are a security analyst."}, {"role": "user", "content": "Analyze the scan results and summarize findings."} ] }' ``` The `model` field maps to the agent name (`claude-code`, `codex`, `opencode`, `gemini`). Defaults to `claude-code` if unrecognized. Only one ACP agent subprocess can run at a time via the API. Concurrent requests return HTTP 409. ## Providers ### OpenAI ```yaml theme={null} llm: llm_providers: - provider: openai base_url: "https://api.openai.com/v1" auth_token: "sk-..." model: gpt-4 ``` ### Anthropic ```yaml theme={null} llm: llm_providers: - provider: anthropic base_url: "https://api.anthropic.com/v1" auth_token: "sk-ant-..." model: claude-3-opus ``` ### Ollama (Local) ```yaml theme={null} llm: llm_providers: - provider: ollama base_url: "http://localhost:11434" model: llama2 ``` ### Azure OpenAI ```yaml theme={null} llm: llm_providers: - provider: azure base_url: "https://your-resource.openai.azure.com" auth_token: "..." model: gpt-4 ``` ### Multiple Providers (Rotation) Configure multiple providers for automatic rotation: ```yaml theme={null} llm: llm_providers: - provider: openai base_url: "https://api.openai.com/v1" auth_token: "sk-..." model: gpt-4 - provider: anthropic base_url: "https://api.anthropic.com/v1" auth_token: "sk-ant-..." model: claude-3-opus - provider: ollama base_url: "http://localhost:11434" model: llama2 ``` ## Workflow Functions Use LLM functions directly in function steps without the full `llm` step type. ### llm\_invoke Simple LLM call with a direct message: ```yaml theme={null} - name: quick-summary type: function function: "llm_invoke('Summarize these findings: ' + readFile('{{Output}}/vulns.txt'))" exports: summary: "{{_result}}" ``` ### llm\_invoke\_custom LLM call with a custom POST body template. Use `{{message}}` as a placeholder: ```yaml theme={null} - name: custom-analysis type: function function: | llm_invoke_custom( 'Analyze this target: {{Target}}', '{"model": "gpt-4", "temperature": 0.5, "messages": [{"role": "user", "content": "{{message}}"}]}' ) ``` ### llm\_conversations Multi-turn conversation using `role:content` format: ```yaml theme={null} - name: conversation type: function function: | llm_conversations( 'system:You are a security analyst.', 'user:What are common web vulnerabilities?', 'assistant:Common web vulnerabilities include SQL injection, XSS, CSRF...', 'user:How do I test for SQL injection?' ) exports: response: "{{_result}}" ``` ## Use Cases ### Vulnerability Analysis ```yaml theme={null} - name: analyze-vulns type: llm messages: - role: system content: | You are a security expert. Analyze vulnerabilities and provide: 1. Risk assessment 2. Impact analysis 3. Remediation steps - role: user content: "{{readFile('{{Output}}/nuclei-results.json')}}" ``` ### Report Generation ```yaml theme={null} - name: generate-report type: llm messages: - role: user content: | Generate a security assessment report for {{target}}. Subdomains found: {{fileLength("{{Output}}/subs.txt")}} Live hosts: {{fileLength("{{Output}}/live.txt")}} Vulnerabilities: {{readFile("{{Output}}/vulns.txt")}} exports: report: "{{generate_report_content}}" - name: save-report type: bash command: echo "{{report}}" > {{Output}}/report.md ``` ### Intelligent Filtering ```yaml theme={null} - name: filter-false-positives type: llm messages: - role: system content: | Analyze these findings and mark false positives. Return JSON: {"valid": [...], "false_positives": [...]} - role: user content: "{{readFile('{{Output}}/findings.json')}}" response_format: type: json_object ``` ### Autonomous Reconnaissance Agent ```yaml theme={null} - name: auto-recon type: agent query: | Perform reconnaissance on {{Target}}: 1. Enumerate subdomains 2. Check for live hosts 3. Identify web technologies 4. Save a summary report to {{Output}}/agent-report.md system_prompt: "You are an autonomous security reconnaissance agent with access to common security tools." max_iterations: 30 memory: max_messages: 40 summarize_on_truncate: true persist_path: "{{Output}}/agent/recon-memory.json" agent_tools: - preset: bash - preset: read_file - preset: save_content - preset: glob - preset: file_exists exports: recon_report: "{{agent_content}}" ``` ### Delegated Code Analysis (Agent-ACP) ```yaml theme={null} - name: code-review type: agent-acp agent: claude-code cwd: "{{Output}}/source" acp_config: write_enabled: true messages: - role: system content: "You are a security code reviewer." - role: user content: | Review the source code in this directory for security vulnerabilities. Focus on OWASP Top 10 issues. Write a report to ./security-review.md exports: review: "{{acp_output}}" ``` ## Best Practices 1. **Use system prompts** for consistent behavior 2. **Limit context size** — summarize large inputs before passing to LLM 3. **Set `max_iterations`** appropriately — higher for complex tasks, lower for simple queries 4. **Enable memory management** for long-running agents to avoid context overflow 5. **Use structured output** when you need to parse the response programmatically 6. **Consider local models** (Ollama) for sensitive data that shouldn't leave your network 7. **Use sub-agents** to decompose complex tasks into specialized subtasks 8. **Add `stop_condition`** when the agent has a clear success criteria 9. **Use `plan_prompt`** for complex tasks that benefit from upfront planning 10. **Use `agent-acp`** when you need full coding agent capabilities (file editing, code generation) — use `agent` when you need fine-grained tool control within the osmedeus ecosystem 11. **Restrict `allowed_paths`** in agent-acp steps to limit file access to the workspace 12. **Keep `write_enabled: false`** (default) in agent-acp unless the agent needs to create files ## Next Steps * [Step Types](../workflows/step-types.md) — LLM and Agent step details * [API Overview](../api/overview.md) — LLM endpoints * [Configuration](../getting-started/configuration.md) — LLM settings # Notification and CDN Configuration Source: https://docs.osmedeus.org/advanced/notification-and-cdn This guide covers advanced configuration options including notification and CDN setup The notification and CDN setup can be modify directly in the settings file. This page is to provide you a quick reference for the most common configurations through CLI and hot to verify your setup. ## Notification Configuration ```bash theme={null} ## Telegram Setup osmedeus config set notification.enabled true osmedeus config set notification.telegram.enabled true osmedeus config set notification.provider telegram osmedeus config set notification.telegram.bot_token "12345:your-token" osmedeus config set notification.telegram.chat_id "-1001234567890" ``` verify your setup by sending a sample message with utility function `osmedeus eval 'notify_telegram("**hola** osmedeus from cli")'` Notification Functions Use `osmedeus func ls noti` to list all available notification functions. ### How to get your telegram token and channel ID Here are a quick guide on how to get your telegram token and channel ID ```bash theme={null} Search for @BotFather and create your bot then grab the token # get channel ID curl "https://api.telegram.org/bot$TELEGRAM_TOKEN/getUpdates" | jq # send a test message curl -X POST "https://api.telegram.org/bot$TELEGRAM_TOKEN/sendMessage" -d chat_id=- -d text="Hello, this is a broadcast to the channel" ``` ## CDN Configuration Use `osmedeus func ls cdn` to list all available CDN functions. Below are some of the functions I usually use: CDN Functions ```bash theme={null} osmedeus eval 'cdn_ls_tree()' # download and upload result osmedeus eval cdn_download("workspace_result.jsonl", "workspace_result.jsonl") osmedeus eval cdn_upload("workspace_result.jsonl", "workspace_result.jsonl") # sync folder to remote osmedeus eval 'cdn_sync_upload("local", "remote")' osmedeus eval 'cdn_ls_tree("targets/")' ``` ### Cloudflare R2 ```bash theme={null} ## CDN Setup for R2 osmedeus config set storage.provider 'r2' osmedeus config set storage.access_key_id '' osmedeus config set storage.bucket '' osmedeus config set storage.secret_access_key '' osmedeus config set storage.endpoint '.r2.cloudflarestorage.com' osmedeus config set storage.presign_expiry '2h' osmedeus config set storage.use_ssl 'true' osmedeus config set storage.enabled 'true' ``` verify your setup by sending a sample message with utility function `osmedeus eval 'cdn_ls_tree()'` ### Google Cloud Storage ```bash theme={null} ## CDN Setup for GCS osmedeus config set storage.provider 'gcs' osmedeus config set storage.access_key_id '' osmedeus config set storage.secret_access_key '' osmedeus config set storage.bucket '' osmedeus config set storage.endpoint 'storage.googleapis.com' osmedeus config set storage.region 'us-east-1' osmedeus config set storage.use_ssl 'true' osmedeus config set storage.presign_expiry '1h' osmedeus config set storage.enabled 'true' ``` verify your setup by sending a sample message with utility function `osmedeus eval 'cdn_ls_tree()'` ## Full Notification and CDN Configuration ```yaml theme={null} # ============================================================================= # Notification Configuration # ============================================================================= # Send notifications when scans complete or find interesting results notification: # Notification provider: "telegram" (future: slack, discord, webhook) provider: telegram # Master switch to enable/disable all notifications enabled: false # Telegram bot settings # Create a bot via @BotFather and get the token # Get your chat ID by messaging @userinfobot telegram: # Bot token from @BotFather bot_token: "" # Chat ID to send messages to (can be user or group) chat_id: 0 # Enable Telegram notifications enabled: false # Channel map for sending to specific channels/groups by name # Use notify_telegram_channel("#name", "message") or send_telegram_file_channel("#name", "/path/to/file") # You can also use numeric chat IDs directly: notify_telegram_channel("-1001234567890", "message") telegram_channel_map: # alerts: -1001234567890 # reports: -1009876543210 # Webhook settings # Configure multiple webhook endpoints for notifications # Webhooks are triggered on workflow events (completed, failed, etc.) webhooks: # Example webhook configuration - url: "https://example.com/webhook" # Enable/disable this webhook enabled: false # Custom HTTP headers (e.g., for authentication) headers: Authorization: "Bearer your-token-here" Content-Type: "application/json" # Request timeout in seconds (default: 30) timeout: 30 # Number of retry attempts on failure (default: 3) retry_count: 3 # Skip TLS certificate verification (default: false) # Set to true for self-signed certificates skip_tls_verify: false # Event filter - only trigger for these events (empty = all events) # Available events: workflow_completed, workflow_failed, workflow_cancelled events: - workflow_completed - workflow_failed # Second webhook example (e.g., Slack incoming webhook) # - url: "https://hooks.slack.com/services/xxx/yyy/zzz" # enabled: false # headers: {} # events: # - workflow_completed # ============================================================================= # Cloud Storage Configuration (Optional) # ============================================================================= # S3-compatible storage for backing up scan results # Supports AWS S3, MinIO, Cloudflare R2, Google Cloud Storage, DigitalOcean Spaces, Oracle OCI storage: # Storage provider: "s3", "minio", "r2", "gcs", "spaces", "oci" # The provider determines endpoint resolution and default settings provider: s3 # Storage endpoint URL (auto-resolved for most providers) # Leave empty to auto-resolve based on provider and region/account_id # Explicit examples: # AWS S3: "s3.us-east-1.amazonaws.com" # MinIO: "localhost:9000" # R2: Will auto-resolve from account_id # GCS: "storage.googleapis.com" # Spaces: Will auto-resolve from region (e.g., "nyc3.digitaloceanspaces.com") # OCI: Will auto-resolve from account_id (namespace) and region endpoint: "" # Access credentials access_key_id: "" secret_access_key: "" # Bucket name for storing results bucket: "" # Cloud region (e.g., us-east-1, eu-west-1) # Required for: s3, spaces, oci region: us-east-1 # Account ID or Namespace (provider-specific) # R2: Your Cloudflare account ID # OCI: Your Object Storage namespace account_id: "" # Use SSL/TLS for connections (default: true for cloud providers) use_ssl: true # Force path-style URLs (auto-configured per provider) # Set true for MinIO, R2, OCI; false for S3, GCS, Spaces path_style: false # Default presigned URL expiry (e.g., "1h", "30m", "24h") # Used by cdnGetPresignedURL when no expiry is specified presign_expiry: "1h" # Enable cloud storage uploads enabled: false ``` # Scheduling & Queue Source: https://docs.osmedeus.org/advanced/scheduling Automate workflow execution with triggers. ## Trigger Types | Type | Description | Use Case | | -------- | ---------------------- | -------------------- | | `cron` | Time-based scheduling | Daily/weekly scans | | `event` | Event-driven execution | React to discoveries | | `watch` | File change detection | Process new data | | `manual` | On-demand only | Placeholder trigger | ## CLI Quick Start The fastest way to set up scheduling, queuing, and webhooks — no YAML required. ### Schedule a Recurring Scan (`--as-cron`) Create a cron schedule directly from the `run` command: ```bash theme={null} # Schedule a module to run daily at 2 AM osmedeus run -m subdomain-enum -t example.com --as-cron '0 2 * * *' # Schedule a flow to run every 6 hours osmedeus run -f full-recon -t example.com --as-cron '0 */6 * * *' # Schedule with additional parameters osmedeus run -m nuclei-scan -t example.com --as-cron '0 0 * * 1' -p threads=20 ``` This creates a schedule record in the database. To activate it, start the server: ```bash theme={null} osmedeus serve ``` List created schedules: ```bash theme={null} osmedeus db ls --table schedules ``` ### Queue a Run for Later (`--queue`) Queue a run for deferred processing instead of executing immediately: ```bash theme={null} # Queue a single target osmedeus run -m subdomain-enum -t example.com --queue # Queue multiple targets osmedeus run -f full-recon -t example.com -t corp.com --queue # Queue from a target file osmedeus run -m nuclei-scan -T targets.txt --queue ``` Queued tasks are stored in the database with status `queued`. Process them with: ```bash theme={null} # Process queued tasks (one at a time) osmedeus worker queue run # Process with concurrency osmedeus worker queue run --concurrency 5 ``` The server also auto-polls for queued tasks when running (every 30s). ### Register a Webhook Trigger (`--as-webhook`) Create a webhook URL that triggers a run on demand: ```bash theme={null} # Register a webhook osmedeus run -m subdomain-enum -t example.com --as-webhook # With an authentication key osmedeus run -f full-recon -t example.com --as-webhook --webhook-auth-key mysecretkey ``` This creates a webhook record and prints the trigger URL. Trigger it with: ```bash theme={null} # GET request (simple trigger) curl http://localhost:8002/osm/api/webhook-runs//trigger # With auth key curl http://localhost:8002/osm/api/webhook-runs//trigger?key=mysecretkey ``` Requires `enable_trigger_via_webhook: true` in `osm-settings.yaml` and a running server. ## Queue Management Full control over queued tasks via the `osmedeus worker queue` subcommands. ### List Queued Tasks ```bash theme={null} # List all queued tasks osmedeus worker queue list # JSON output osmedeus worker queue list --json ``` ### Create Queued Tasks An alternative to `--queue` on `osmedeus run`: ```bash theme={null} # Queue a flow osmedeus worker queue new -f full-recon -t example.com # Queue a module with multiple targets osmedeus worker queue new -m subdomain-enum -t example.com -t corp.com # Queue from a target file with params osmedeus worker queue new -m nuclei-scan -T targets.txt -p threads=20 ``` ### Process Queued Tasks ```bash theme={null} # Start processing (polls DB every 5s) osmedeus worker queue run # With concurrency osmedeus worker queue run --concurrency 5 # With a Redis connection for dual-source polling osmedeus worker queue run --redis-url redis://localhost:6379 ``` The queue runner polls the database every 5 seconds. If Redis is configured, it also listens via `BRPOP` for lower latency. ### Server-Side Queue Polling The server (`osmedeus serve`) automatically polls for queued tasks every 30 seconds. Disable with: ```bash theme={null} osmedeus serve --no-queue-polling ``` ## Webhook Management ### List Registered Webhooks ```bash theme={null} osmedeus worker webhooks ``` Displays a table of all registered webhook triggers with their UUIDs, workflow, target, trigger URL, and auth key. ### Triggering Webhooks Webhooks can be triggered via GET or POST: ```bash theme={null} # Simple GET trigger curl http://localhost:8002/osm/api/webhook-runs//trigger # POST with overrides (target, flow, or module) curl -X POST http://localhost:8002/osm/api/webhook-runs//trigger \ -H "Content-Type: application/json" \ -d '{"target": "other.com"}' # Override the workflow curl -X POST http://localhost:8002/osm/api/webhook-runs//trigger \ -H "Content-Type: application/json" \ -d '{"target": "other.com", "module": "nuclei-scan"}' ``` POST body fields (all optional — defaults come from the registered webhook): | Field | Description | | -------- | ------------------------------- | | `target` | Override the target | | `flow` | Override with a flow workflow | | `module` | Override with a module workflow | ### Authentication If a webhook was registered with `--webhook-auth-key`, the key must be provided as a query parameter: ```bash theme={null} curl http://localhost:8002/osm/api/webhook-runs//trigger?key=mysecretkey ``` Requests without the correct key receive a `401 Unauthorized` response. ### Webhook Configuration Webhook triggering must be enabled in `osm-settings.yaml`: ```yaml theme={null} server: enable_trigger_via_webhook: true ``` ## Server Flags Relevant flags for `osmedeus serve`: | Flag | Description | | --------------------- | -------------------------------------- | | `--no-schedule` | Disable the cron/watch/event scheduler | | `--no-queue-polling` | Disable background queue task polling | | `--no-event-receiver` | Disable automatic event receiver | | `--no-hot-reload` | Disable config hot reload | ## Cron Triggers Execute workflows on a schedule using cron expressions. ### Workflow Definition ```yaml theme={null} kind: module name: scheduled-scan triggers: - name: daily-scan on: cron schedule: "0 2 * * *" # Daily at 2 AM enabled: true params: - name: target required: true steps: - name: scan type: bash command: nuclei -u {{target}} ``` ### Cron Expression Format ``` ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday = 0) │ │ │ │ │ * * * * * ``` ### Common Schedules | Expression | Description | | ---------------- | ------------------- | | `0 * * * *` | Every hour | | `0 2 * * *` | Daily at 2 AM | | `0 0 * * 0` | Weekly on Sunday | | `0 0 1 * *` | Monthly on the 1st | | `*/15 * * * *` | Every 15 minutes | | `0 9-17 * * 1-5` | Hourly 9-5 weekdays | ## Event Triggers Execute workflows in response to events. ### Workflow Definition ```yaml theme={null} kind: module name: on-asset-discovered triggers: - name: new-asset on: event event: topic: "assets.new" filters: - "event.source == 'subfinder'" - "event.data.type == 'subdomain'" input: type: event_data field: "url" name: target enabled: true steps: - name: probe type: bash command: httpx -u {{target}} ``` ### Event Topics Topics support glob patterns for flexible matching: ```yaml theme={null} event: topic: "assets.*" # Matches assets.new, assets.updated topic: "*.new" # Matches assets.new, vulns.new topic: "*" # Matches all topics ``` | Topic | Description | | --------------------- | ----------------------- | | `run.started` | Workflow run started | | `run.completed` | Workflow run completed | | `run.failed` | Workflow run failed | | `step.completed` | Step completed | | `assets.new` | New asset discovered | | `vulnerabilities.new` | New vulnerability found | ### Event Filters JavaScript expressions to filter events: ```yaml theme={null} filters: - "event.source == 'nuclei'" - "event.data.severity == 'critical'" - "event.workspace == 'example.com'" ``` For advanced filtering with utility functions, use `filter_functions`: ```yaml theme={null} event: topic: "assets.new" filters: - "event.source == 'httpx'" filter_functions: - "contains(event.data.url, '/api/')" - "!ends_with(event.data.url, '.js')" ``` See [Event-Driven Triggers](event-driven.mdx) for available filter functions. ### Event Input Mapping Two syntaxes are supported: **New exports-style syntax (recommended):** ```yaml theme={null} input: target: event_data.url source: event.source description: trim(event_data.desc) ``` **Legacy syntax:** ```yaml theme={null} input: type: event_data field: "target" # Field in event data name: target # Workflow parameter name ``` ### Event Template Variables Event-triggered workflows have access to these template variables: | Variable | Description | | -------------------- | ------------------------ | | `{{EventEnvelope}}` | Full JSON event envelope | | `{{EventTopic}}` | Event topic | | `{{EventSource}}` | Event source | | `{{EventDataType}}` | Data type | | `{{EventTimestamp}}` | Event timestamp | ## Watch Triggers Execute workflows when files change. Uses fsnotify for instant inotify-based file system notifications. ### Workflow Definition ```yaml theme={null} kind: module name: process-new-targets triggers: - name: new-targets on: watch path: "/data/targets/" debounce: "500ms" # Optional: debounce rapid changes input: type: file_path name: target_file enabled: true steps: - name: process type: foreach input: "{{target_file}}" variable: target step: type: bash command: scan {{target}} ``` ### Watch Configuration | Field | Type | Description | | ---------- | ------ | ------------------------------------------------------ | | `path` | string | Directory or file path to watch | | `debounce` | string | Optional debounce duration (e.g., "500ms", "1s", "2s") | ### Debounce When files are modified rapidly (e.g., during a write operation), multiple events may fire. Use `debounce` to consolidate rapid changes into a single trigger: ```yaml theme={null} triggers: - name: watch-logs on: watch path: "/var/log/scan-results/" debounce: "1s" # Wait 1 second after last change before triggering enabled: true ``` The debounce timer resets on each new file event. The workflow only triggers after the specified duration has passed without new events. ### Watch Events | Event | Description | | -------- | ---------------- | | `create` | New file created | | `modify` | File modified | | `delete` | File deleted | | `rename` | File renamed | ## API Management ### List Schedules ```bash theme={null} curl http://localhost:8002/osm/api/schedules \ -H "Authorization: Bearer $TOKEN" ``` ### Create Schedule ```bash theme={null} curl -X POST http://localhost:8002/osm/api/schedules \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "daily-scan", "workflow_name": "full-recon", "trigger_type": "cron", "schedule": "0 2 * * *", "input_config": { "target": "example.com" }, "is_enabled": true }' ``` ### Enable/Disable ```bash theme={null} # Enable curl -X POST http://localhost:8002/osm/api/schedules/sched-123/enable \ -H "Authorization: Bearer $TOKEN" # Disable curl -X POST http://localhost:8002/osm/api/schedules/sched-123/disable \ -H "Authorization: Bearer $TOKEN" ``` ### Trigger Manually ```bash theme={null} curl -X POST http://localhost:8002/osm/api/schedules/sched-123/trigger \ -H "Authorization: Bearer $TOKEN" ``` ### Delete Schedule ```bash theme={null} curl -X DELETE http://localhost:8002/osm/api/schedules/sched-123 \ -H "Authorization: Bearer $TOKEN" ``` ### Webhook API Endpoints List registered webhooks (authenticated): ```bash theme={null} curl http://localhost:8002/osm/api/webhook-runs/list \ -H "Authorization: Bearer $TOKEN" ``` Trigger a webhook (unauthenticated, unless `webhook_auth_key` is set): ```bash theme={null} # GET curl http://localhost:8002/osm/api/webhook-runs//trigger # POST with overrides curl -X POST http://localhost:8002/osm/api/webhook-runs//trigger \ -H "Content-Type: application/json" \ -d '{"target": "other.com", "module": "nuclei-scan"}' ``` ## Combined Triggers A workflow can have multiple triggers: ```yaml theme={null} kind: module name: flexible-scan triggers: # Run daily - name: daily on: cron schedule: "0 2 * * *" enabled: true # Run on new assets - name: on-asset on: event event: topic: "assets.new" input: type: event_data field: "url" name: target enabled: true # Manual trigger placeholder - name: manual on: manual enabled: true params: - name: target required: true steps: - name: scan type: bash command: scan {{target}} ``` ## Event Emission Emit events from workflows: ```yaml theme={null} - name: scan type: bash command: nuclei -u {{target}} -o {{Output}}/vulns.json on_success: - action: emit_event topic: "scan.completed" data: target: "{{target}}" results: "{{Output}}/vulns.json" ``` ## Database Storage Schedules are stored in the database: ```sql theme={null} -- schedules table id, name, workflow_name, workflow_path, trigger_type, schedule, event_topic, watch_path, input_config, is_enabled, last_run, next_run, run_count, created_at, updated_at ``` Query schedules: ```bash theme={null} osmedeus db query "SELECT * FROM schedules WHERE is_enabled = true" ``` ## Best Practices 1. **Use descriptive trigger names** 2. **Start with disabled triggers** during testing 3. **Set reasonable intervals** to avoid overload 4. **Use event filters** to reduce noise 5. **Monitor trigger execution** via event logs 6. **Combine with distributed mode** for scale ## Troubleshooting ### Schedule not running ```bash theme={null} # Check if enabled curl http://localhost:8002/osm/api/schedules/sched-123 # Check next_run time # Ensure server is running ``` ### Events not triggering ```bash theme={null} # Check event logs curl "http://localhost:8002/osm/api/event-logs?topic=assets.new" # Verify filter expressions ``` ### Watch not detecting changes ```bash theme={null} # Verify path exists # Check file permissions # Ensure pattern matches ``` ## Next Steps * [Distributed Execution](distributed.md) - Scale with workers * [Server CLI](../cli/server.md) - Server setup * [API Overview](../api/overview.md) - Schedule endpoints # Snapshots Source: https://docs.osmedeus.org/advanced/snapshots Export and import workspaces as compressed ZIP archives. ## Overview Snapshots enable: * Backup and restore workspaces * Share scan results * Migrate between machines * Archive completed assessments ## Export ### CLI ```bash theme={null} # Export workspace osmedeus snapshot export example.com # Custom output path osmedeus snapshot export example.com -o /backups/example.zip # Export to specific directory osmedeus snapshot export example.com -o ~/archives/ ``` ### API ```bash theme={null} curl -X POST http://localhost:8002/osm/api/snapshots/export \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"workspace": "example.com"}' ``` ### Contents Exported ZIP includes: ``` example.com-20240115-143022.zip ├── workspace/ # All workspace files │ ├── subdomain-enum/ │ ├── http-probe/ │ └── ... ├── metadata.json # Export metadata └── database.json # Database records ├── assets ├── vulnerabilities └── runs ``` ## Import ### CLI ```bash theme={null} # Import from local file osmedeus snapshot import ~/backup.zip # Import from URL osmedeus snapshot import https://example.com/workspace.zip # Force overwrite existing osmedeus snapshot import ~/backup.zip --force # Files only (skip database) osmedeus snapshot import ~/backup.zip --skip-db ``` ### API ```bash theme={null} # Import from URL curl -X POST http://localhost:8002/osm/api/snapshots/import \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"source": "https://example.com/backup.zip"}' # Import with options curl -X POST http://localhost:8002/osm/api/snapshots/import \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "source": "/path/to/backup.zip", "force": true, "skip_db": false }' ``` ## List Snapshots ### CLI ```bash theme={null} osmedeus snapshot list ``` ### API ```bash theme={null} curl http://localhost:8002/osm/api/snapshots \ -H "Authorization: Bearer $TOKEN" ``` ### Output ``` Available Snapshots: ┌────────────────────────────────────────────┬────────────┬─────────────────────┐ │ Filename │ Size │ Created │ ├────────────────────────────────────────────┼────────────┼─────────────────────┤ │ example.com-20240115-143022.zip │ 12.5 MB │ 2024-01-15 14:30:22 │ │ target.org-20240114-091500.zip │ 8.2 MB │ 2024-01-14 09:15:00 │ └────────────────────────────────────────────┴────────────┴─────────────────────┘ ``` ## Download ```bash theme={null} # Via API curl http://localhost:8002/osm/api/snapshots/download/example.com-backup.zip \ -H "Authorization: Bearer $TOKEN" \ -o local-backup.zip ``` ## Delete ```bash theme={null} curl -X DELETE http://localhost:8002/osm/api/snapshots/example.com-backup.zip \ -H "Authorization: Bearer $TOKEN" ``` ## Storage Location Default: `~/osmedeus-base/snapshots/` Configure in `osm-settings.yaml`: ```yaml theme={null} environments: storages: "{{base_folder}}/snapshots" ``` ## Use Cases ### Backup Before Destructive Scan ```bash theme={null} # Backup current state osmedeus snapshot export example.com # Run potentially destructive scan osmedeus run -f aggressive-scan -t example.com # Restore if needed osmedeus snapshot import ~/osmedeus-base/snapshots/example.com-*.zip --force ``` ### Share Results ```bash theme={null} # Export workspace osmedeus snapshot export client-assessment -o /tmp/results.zip # Share via cloud storage aws s3 cp /tmp/results.zip s3://bucket/results.zip ``` ### Migrate to Another Machine ```bash theme={null} # On source machine osmedeus snapshot export example.com -o /tmp/workspace.zip scp /tmp/workspace.zip user@target:/tmp/ # On target machine osmedeus snapshot import /tmp/workspace.zip ``` ### Archive Completed Projects ```bash theme={null} # Export and compress osmedeus snapshot export project-2024 -o /archives/project-2024.zip # Remove workspace rm -rf ~/osmedeus-base/workspaces/project-2024 # Restore later if needed osmedeus snapshot import /archives/project-2024.zip ``` ## Metadata Each snapshot includes `metadata.json`: ```json theme={null} { "version": "1.0", "workspace": "example.com", "created_at": "2024-01-15T14:30:22Z", "osmedeus_version": "2.0.0", "files_count": 156, "total_size": 13107200, "records": { "assets": 423, "vulnerabilities": 15, "runs": 3 } } ``` ## Database Records The `database.json` contains: ```json theme={null} { "assets": [ { "id": 1, "workspace": "example.com", "asset_value": "sub.example.com", "url": "https://sub.example.com", "status_code": 200, ... } ], "vulnerabilities": [ { "id": 1, "workspace": "example.com", "title": "SQL Injection", "severity": "high", ... } ], "runs": [ { "id": "run-abc123", "workflow_name": "full-recon", "target": "example.com", ... } ] } ``` ## Import Options ### Force Overwrite ```bash theme={null} osmedeus snapshot import backup.zip --force ``` * Deletes existing workspace * Removes existing database records * Imports fresh from snapshot ### Skip Database ```bash theme={null} osmedeus snapshot import backup.zip --skip-db ``` * Only imports files * Preserves existing database records * Useful for file-only updates ## Cloud Storage ### Export to S3 ```bash theme={null} # Export locally osmedeus snapshot export example.com -o /tmp/backup.zip # Upload to S3 aws s3 cp /tmp/backup.zip s3://bucket/snapshots/ # Or configure storage in settings storage: enabled: true provider: s3 bucket: osmedeus-snapshots ``` ### Import from S3 ```bash theme={null} # Generate presigned URL URL=$(aws s3 presign s3://bucket/snapshots/backup.zip) # Import from URL osmedeus snapshot import "$URL" ``` ## Best Practices 1. **Regular backups** - Schedule snapshot exports 2. **Verify after import** - Check file integrity 3. **Use descriptive names** - Include date and purpose 4. **Secure storage** - Encrypt sensitive snapshots 5. **Clean old snapshots** - Manage storage usage ## Troubleshooting ### Import fails ```bash theme={null} # Check file integrity unzip -t backup.zip # Try with verbose output osmedeus snapshot import backup.zip -v ``` ### Missing files after import ```bash theme={null} # Check if skip-db was used # Re-import with database osmedeus snapshot import backup.zip ``` ### Large snapshot size ```bash theme={null} # Check workspace contents du -sh ~/osmedeus-base/workspaces/example.com/* # Clean unnecessary files before export rm -rf ~/osmedeus-base/workspaces/example.com/temp/ ``` ## Next Steps * [Snapshot CLI](../cli/snapshot.md) - CLI reference * [API Overview](../api/overview.md) - Snapshot endpoints * [Workspaces](../concepts/workflows.md) - Workspace structure # Writing Your First Workflow Source: https://docs.osmedeus.org/advanced/writing-your-first-workflow YAML-based workflow definitions and execution You can find comprehensive examples of all available workflow types and steps in the [Workflow test suites here](https://github.com/j3ssie/osmedeus/tree/main/test/testdata/workflows). You can also use the skills provided at [osmedeus/osmedeus-skills](https://github.com/osmedeus/osmedeus-skills). These can help your AI agent generate workflows automatically for you. This guide walks you through creating workflows in Osmedeus, from basic concepts to advanced patterns. ## Workflow Kinds Osmedeus supports two workflow kinds: | Kind | Purpose | | -------- | -------------------------------- | | `module` | Single execution unit with steps | | `flow` | Orchestrates multiple modules | ## Basic Structure ### Module Workflow ```yaml theme={null} name: my-first-workflow kind: module description: A simple workflow example tags: example,tutorial params: - name: custom_param required: false default: "default_value" steps: - name: hello-world type: bash command: echo "Hello, {{Target}}!" ``` ### Flow Workflow ```yaml theme={null} name: my-flow kind: flow description: Orchestrates multiple modules modules: - name: subdomain-enum path: modules/subdomain-enum.yaml - name: port-scan path: modules/port-scan.yaml depends_on: - subdomain-enum ``` ## Step Types ### bash - Execute Shell Commands ```yaml theme={null} # Single command - name: simple-command type: bash command: echo "Hello {{Target}}" # Multiple sequential commands - name: multiple-commands type: bash commands: - mkdir -p {{Output}}/results - echo "{{Target}}" > {{Output}}/target.txt # Parallel commands - name: parallel-commands type: bash parallel_commands: - 'curl -s https://api1.example.com' - 'curl -s https://api2.example.com' # Structured arguments (for tools like nuclei) - name: nuclei-scan type: bash command: nuclei speed_args: '-c {{threads}}' config_args: '-t /templates' input_args: '-l {{Output}}/urls.txt' output_args: '-o {{Output}}/nuclei.json' ``` ### function - JavaScript Utility Functions ```yaml theme={null} # Single function - name: check-file type: function function: 'fileExists("{{Output}}/results.txt")' # Multiple functions - name: process-results type: function functions: - 'log_info("Processing results...")' - 'var count = fileLength("{{Output}}/results.txt")' - 'log_info("Found " + count + " results")' # Parallel functions - name: parallel-logging type: function parallel_functions: - 'log_info("Task A")' - 'log_info("Task B")' ``` ### parallel-steps - Run Steps Concurrently ```yaml theme={null} - name: parallel-recon type: parallel-steps parallel_steps: - name: subfinder type: bash command: subfinder -d {{Target}} -o {{Output}}/subfinder.txt - name: assetfinder type: bash command: assetfinder {{Target}} > {{Output}}/assetfinder.txt - name: amass type: bash command: amass enum -passive -d {{Target}} -o {{Output}}/amass.txt ``` ### foreach - Loop Over Input ```yaml theme={null} - name: scan-subdomains type: foreach input: "{{Output}}/subdomains.txt" variable: subdomain threads: 10 step: name: httpx-probe type: bash command: 'httpx -u [[subdomain]] -silent' ``` **Note:** Use `[[variable]]` syntax inside foreach loops to avoid template conflicts. ### http - Make HTTP Requests ```yaml theme={null} - name: api-call type: http url: "https://api.example.com/scan" method: POST headers: Content-Type: application/json Authorization: "Bearer {{api_token}}" request_body: | { "target": "{{Target}}", "options": {"deep": true} } exports: response_data: "{{response.body}}" ``` ### llm - AI-Powered Analysis ```yaml theme={null} - name: ai-analysis type: llm messages: - role: system content: "You are a security analyst." - role: user content: "Analyze the scan results for {{Target}}" llm_config: model: gpt-4 max_tokens: 1000 temperature: 0.7 exports: analysis: "{{llm_step_content}}" ``` ### agent - Agentic LLM Execution ```yaml theme={null} - name: analyze-target type: agent query: "Enumerate subdomains of {{Target}} and summarize findings." system_prompt: "You are a security reconnaissance agent." max_iterations: 10 agent_tools: - preset: bash - preset: read_file - preset: save_content memory: max_messages: 30 persist_path: "{{Output}}/agent/conversation.json" exports: findings: "{{agent_content}}" ``` The agent step type creates an autonomous tool-calling loop. The agent receives a task, plans its approach, calls tools iteratively, and produces a final answer. Key fields: * `query` — task prompt for the agent * `max_iterations` — maximum tool-calling loop iterations (required) * `agent_tools` — list of preset or custom tools (e.g., `bash`, `read_file`, `save_content`, `grep_regex`, `http_get`) * `memory` — conversation memory configuration * `exports` — use `{{agent_content}}` for the final response text See [Step Types - agent](../workflows/step-types#agent) for the full reference. ## Template Variables ### Built-in Variables | Variable | Description | | --------------------------- | ----------------------------------------------- | | `{{Target}}` | Current target | | `{{Output}}` | Output directory for this run | | `{{BaseFolder}}` | Osmedeus installation directory | | `{{Binaries}}` | Binary tools directory | | `{{Data}}` | Data directory (wordlists, etc.) | | `{{Workflows}}` | Workflows directory | | `{{Workspaces}}` | Workspaces directory | | `{{threads}}` | Thread count based on tactic | | `{{Version}}` | Osmedeus version | | `{{PlatformOS}}` | Operating system (`linux`, `darwin`, `windows`) | | `{{PlatformArch}}` | CPU architecture (`amd64`, `arm64`) | | `{{PlatformInDocker}}` | `"true"` if running in Docker | | `{{PlatformInKubernetes}}` | `"true"` if running in Kubernetes | | `{{PlatformCloudProvider}}` | Cloud provider (`aws`, `gcp`, `azure`, `local`) | ### Foreach Loop Variables Use double brackets `[[variable]]` inside foreach loops: ```yaml theme={null} - name: process-items type: foreach input: "{{Output}}/items.txt" variable: item step: type: bash command: 'process [[item]] --output {{Output}}/[[item]].json' ``` ## Exports and Variable Passing Pass data between steps using exports: ```yaml theme={null} - name: count-results type: bash command: wc -l {{Output}}/results.txt | awk '{print $1}' exports: result_count: "output" # Special: captures stdout - name: log-count type: function function: 'log_info("Found {{result_count}} results")' ``` ## Decision Routing Branch workflow execution based on conditions. Decisions support two modes: **switch/case** (exact string matching) and **conditions** (boolean expressions). ### Switch/Case Mode Match a variable's value against exact strings: ```yaml theme={null} - name: detect-type type: bash command: 'detect-target-type {{Target}}' exports: target_type: "output" decision: switch: "{{target_type}}" cases: "domain": goto: subdomain-enum "ip": goto: port-scan "url": goto: web-scan default: goto: generic-recon - name: subdomain-enum type: bash command: subfinder -d {{Target}} - name: port-scan type: bash command: nmap {{Target}} # Use goto: _end to terminate workflow early ``` ### Inline Actions in Cases Each case can run inline commands or functions instead of (or in addition to) a `goto`. When combined with `goto`, inline actions execute first, then the jump happens. **Available case fields:** | Field | Type | Description | | ----------- | --------- | ---------------------------------------------- | | `goto` | string | Jump to a step by name, or `_end` to terminate | | `command` | string | Run a single bash command inline | | `commands` | string\[] | Run multiple bash commands in sequence | | `function` | string | Execute a single utility function | | `functions` | string\[] | Execute multiple utility functions in sequence | ```yaml theme={null} - name: setup-scan type: bash command: echo "Setting up scan" exports: setup_mode: "{{scan_mode}}" decision: switch: "{{setup_mode}}" cases: "quick": functions: - "log_info('Configuring quick scan')" - "log_info('Quick scan configured')" "deep": functions: - "log_info('Configuring deep scan')" - "log_info('Deep scan configured')" goto: deep-scan-step "custom": command: echo "Running custom setup" "multi": commands: - echo "Step 1: prepare environment" - echo "Step 2: configure tools" default: function: "log_info('Using default scan mode')" ``` ### Conditions Mode Use JavaScript boolean expressions for more flexible routing. All matching conditions execute (no short-circuit), and the last matching `goto` wins. ```yaml theme={null} - name: check-results type: bash command: echo "Checking results" decision: conditions: - if: "{{enable_extra}} && {{target}} != ''" function: "log_info('Extra scanning enabled')" goto: extra-scan - if: "file_length('{{Output}}/results.txt') > 100" functions: - "log_info('Large result set detected')" - "log_info('Switching to batch processing')" goto: batch-process - if: "file_exists('{{Output}}/errors.log')" command: echo "Errors detected, reviewing..." ``` Conditions support template variables, function calls, and standard JavaScript operators. ## Handlers (on\_success / on\_error) ```yaml theme={null} - name: critical-scan type: bash command: 'nuclei -u {{Target}}' on_success: - action: log message: "Scan completed for {{Target}}" - action: notify notify: "Scan finished: {{Target}}" - action: export name: scan_status value: "success" on_error: - action: log message: "Scan failed for {{Target}}" - action: continue # Continue despite error # Or: action: abort to stop workflow ``` ## Workflow Hooks Hooks let you run steps before and after the main workflow execution. Use them for setup, cleanup, notifications, or result post-processing. ```yaml theme={null} name: recon-with-hooks kind: module description: Reconnaissance with setup and cleanup hooks hooks: pre_scan_steps: - name: setup-workspace type: bash commands: - mkdir -p {{Output}}/results - echo "Scan started at $(date)" > {{Output}}/scan.log - name: notify-start type: function function: | generate_event("{{Workspace}}", "scan.started", "workflow", "status", "{{Target}}") post_scan_steps: - name: generate-report type: function function: | convert_sarif_to_markdown("{{Output}}/results.sarif", "{{Output}}/report.md") - name: notify-complete type: function function: | generate_event("{{Workspace}}", "scan.completed", "workflow", "status", "{{Target}}") - name: cleanup-temp type: bash command: rm -rf {{Output}}/tmp steps: - name: run-scan type: bash command: nuclei -u {{Target}} -sarif-export {{Output}}/results.sarif ``` ### Hook Execution Order ``` pre_scan_steps → steps (main workflow) → post_scan_steps ``` * **pre\_scan\_steps** run before any main steps execute * **post\_scan\_steps** run after all main steps complete * Both support all step types (bash, function, parallel-steps, foreach, etc.) * Hook steps have access to the same template variables as main steps ### Flow-Level Hooks Hooks also work on flows. They run once around the entire flow, not per module: ```yaml theme={null} name: full-recon kind: flow description: Full recon flow with hooks hooks: pre_scan_steps: - name: pre-flight-check type: function function: 'log_info("Starting flow for " + "{{Target}}")' post_scan_steps: - name: aggregate-results type: bash command: cat {{Output}}/*/findings.txt | sort -u > {{Output}}/all-findings.txt modules: - name: subdomain-enum path: modules/subdomain-enum.yaml - name: port-scan path: modules/port-scan.yaml ``` ## Runner Configuration ### Host Runner (Default) ```yaml theme={null} runner: host # Runs locally, this is the default ``` ### Docker Runner ```yaml theme={null} runner: docker runner_config: image: ubuntu:22.04 env: API_KEY: "{{api_key}}" volumes: - "{{Output}}:/output" network: host persistent: false ``` ### SSH Runner ```yaml theme={null} runner: ssh runner_config: host: 192.168.1.100 port: 22 user: scanner key_file: ~/.ssh/id_rsa ``` ### Per-Step Runner Override ```yaml theme={null} - name: docker-step type: bash step_runner: docker step_runner_config: image: projectdiscovery/nuclei:latest command: 'nuclei -u {{Target}}' ``` ## Complete Example ```yaml theme={null} name: basic-recon kind: module description: Basic reconnaissance workflow tags: recon,subdomain,fast params: - name: threads default: "10" steps: - name: setup type: bash commands: - mkdir -p {{Output}}/subdomains - mkdir -p {{Output}}/urls - name: passive-enum type: parallel-steps parallel_steps: - name: subfinder type: bash command: subfinder -d {{Target}} -silent -o {{Output}}/subdomains/subfinder.txt - name: assetfinder type: bash command: assetfinder --subs-only {{Target}} > {{Output}}/subdomains/assetfinder.txt - name: merge-results type: bash commands: - cat {{Output}}/subdomains/*.txt | sort -u > {{Output}}/subdomains.txt exports: subdomain_count: "output" - name: probe-http type: foreach input: "{{Output}}/subdomains.txt" variable: sub threads: "{{threads}}" step: name: httpx type: bash command: 'httpx -u [[sub]] -silent >> {{Output}}/urls/live.txt' - name: summary type: function functions: - 'var total = fileLength("{{Output}}/subdomains.txt")' - 'var live = fileLength("{{Output}}/urls/live.txt")' - 'log_info("Found " + total + " subdomains, " + live + " live hosts")' ``` ## Running Your Workflow ```bash theme={null} # Run a module workflow osmedeus run -m basic-recon -t example.com # Run with custom parameters osmedeus run -m basic-recon -t example.com --params 'threads=20' # Dry run (preview without executing) osmedeus run -m basic-recon -t example.com --dry-run # Run with verbose output osmedeus run -m basic-recon -t example.com -v ``` ## Next Steps * See [CLI References](cli-references.md) for all command options * See [Extending Osmedeus](extending-osmedeus.md) to add custom step types * See [Advanced Configuration](advanced-configuration.md) for API keys and storage setup # Osmedeus API Documentation Source: https://docs.osmedeus.org/api-references/index RESTful API reference for managing security automation workflows # Osmedeus API Documentation ## Overview The Osmedeus API provides a RESTful interface for managing security automation workflows, runs, and distributed task execution. **Base URL:** `http://localhost:8002` **Default Port:** `8002` ## Authentication Most API endpoints require authentication. Two methods are supported: 1. **JWT Token**: Obtain a token via the login endpoint, then include it in requests using the `Authorization: Bearer ` header. 2. **API Key**: Use a static API key via the `x-osm-api-key` header. Configure in `~/osmedeus-base/osm-settings.yaml` under `server.auth_api_key`. See [Authentication](/api-references/authentication) for details. ## API Reference | Category | Description | | -------------------------------------------------- | ----------------------------------------------------- | | [Public Endpoints](/api-references/public) | Server info, health checks, Swagger docs | | [Authentication](/api-references/authentication) | Login, logout, and JWT token management | | [Workflows](/api-references/workflows) | List, view, and refresh workflows | | [Runs](/api-references/runs) | Create and manage workflow executions | | [File Uploads](/api-references/uploads) | Upload target files and workflows | | [Snapshots](/api-references/snapshots) | Export and import workspace snapshots | | [Workspaces](/api-references/workspaces) | List and manage workspaces | | [Artifacts](/api-references/artifacts) | List and download output artifacts | | [Assets](/api-references/assets) | View discovered assets | | [Vulnerabilities](/api-references/vulnerabilities) | View and manage vulnerabilities | | [Event Logs](/api-references/event-logs) | View execution event logs | | [Step Results](/api-references/steps) | Query step execution results | | [Functions](/api-references/functions) | Execute and list utility functions | | [System Statistics](/api-references/system) | Get aggregated system stats | | [Settings](/api-references/settings) | Manage server configuration | | [Database](/api-references/database) | Database management and cleanup | | [Installation](/api-references/install) | Install binaries and workflows | | [Schedules](/api-references/schedules) | Manage scheduled workflows | | [Event Receiver](/api-references/event-receiver) | Event-triggered workflows | | [Distributed Mode](/api-references/distributed) | Worker and task management | | [LLM API](/api-references/llm) | Large Language Model API | | [Reference](/api-references/reference) | Error codes, pagination, cron expressions, step types | ## Quick Start ```bash theme={null} # Get server info (no auth required) curl http://localhost:8002/server-info # Login and get token export TOKEN=$(curl -s -X POST http://localhost:8002/osm/api/login \ -H "Content-Type: application/json" \ -d '{"username": "osmedeus", "password": "admin"}' | jq -r '.token') # List workflows curl http://localhost:8002/osm/api/workflows \ -H "Authorization: Bearer $TOKEN" # Start a scan curl -X POST http://localhost:8002/osm/api/runs \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"flow": "subdomain-enum", "target": "example.com"}' ``` # Architecture Overview Source: https://docs.osmedeus.org/architecture/index Technical architecture overview of the Osmedeus workflow engine # Architecture Overview Osmedeus is a workflow engine for security automation. It executes YAML-defined workflows with support for multiple execution environments, distributed processing, and extensive customization. ## Layered Architecture ``` +-------------------------------------------------------------+ | CLI / REST API | | (pkg/cli, pkg/server) | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | Executor | | (internal/executor) | | Coordinates workflow execution and manages state | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | Step Dispatcher | | | | +----------+ +----------+ +----------+ +----------+ | | | Bash | | Function | | Parallel | | Foreach | | | | Executor | | Executor | | Executor | | Executor | | | +----------+ +----------+ +----------+ +----------+ | | | | +----------+ +----------+ +----------+ +----------+ | | | Remote | | HTTP | | LLM | | Fragment | | | | Bash | | Executor | | Executor | | Executor | | | +----------+ +----------+ +----------+ +----------+ | | | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | Runner | | (internal/runner) | | | | +------------+ +------------+ +------------+ | | | Host | | Docker | | SSH | | | | Runner | | Runner | | Runner | | | +------------+ +------------+ +------------+ | | | +-------------------------------------------------------------+ ``` ## Core Packages | Package | Purpose | | -------------------- | ------------------------------------------------------------------------- | | `internal/core` | Type definitions: Workflow, Step, Trigger, RunnerConfig, ExecutionContext | | `internal/parser` | YAML parsing, validation, and caching (Loader) | | `internal/executor` | Workflow execution engine with step dispatching | | `internal/runner` | Execution environments implementing Runner interface | | `internal/template` | `{{Variable}}` interpolation engine (Engine, ShardedEngine) | | `internal/functions` | Utility functions via Goja JavaScript runtime pool | | `internal/scheduler` | Cron, event, and file-watch triggers | | `internal/database` | SQLite/PostgreSQL via Bun ORM | | `internal/linter` | Workflow validation and linting | | `pkg/cli` | Cobra CLI commands | | `pkg/server` | Fiber REST API | | `internal/snapshot` | Workspace export/import as ZIP archives | | `internal/installer` | Binary installation (direct-fetch and Nix) | | `internal/state` | Run state export for debugging | | `internal/updater` | Self-update via GitHub releases | ## Workflow Execution Flow ``` +-------------------------------------------------------------+ | 1. CLI parses arguments | | osmedeus run -f general -t example.com | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | 2. Load config from ~/osmedeus-base/osm-settings.yaml | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | 3. Parser loads YAML workflow | | - Validates schema | | - Resolves includes (fragments) | | - Caches in Loader | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | 4. Executor initializes context | | - Injects built-in variables (Target, Output, etc.) | | - Loads params with defaults | | - Creates execution context | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | 5. For each step: | | +-----------------------------------------------------+ | | | a. Check depends_on (wait for dependencies) | | | | b. Evaluate pre_condition | | | | c. Render templates | | | | d. Dispatch to appropriate executor | | | | e. Execute via runner | | | | f. Capture output and exports | | | | g. Evaluate decision routing | | | | h. Handle on_success/on_error | | | +-----------------------------------------------------+ | +-------------------------------------------------------------+ | v +-------------------------------------------------------------+ | 6. Export results | | - Save run state | | - Update database | | - Generate artifacts | +-------------------------------------------------------------+ ``` ## Step Type Routing The Step Dispatcher routes steps to the appropriate executor based on type: ``` +-------------------------------------------------------------+ | Step Dispatcher | | | | Step Type Executor | | ------------------------------------------------- | | "bash" ---------> BashExecutor --> Runner | | "function" ---------> FunctionExecutor --> Goja | | "parallel-steps" ---------> ParallelExecutor | | "foreach" ---------> ForeachExecutor | | "remote-bash" ---------> RemoteBashExecutor --> SSH/Docker | | "http" ---------> HTTPExecutor | | "llm" ---------> LLMExecutor | | "fragment-step" ---------> FragmentStepExecutor --> Inline | | | +-------------------------------------------------------------+ ``` ## Key Types ### WorkflowKind ```go theme={null} const ( KindModule WorkflowKind = "module" // Single unit workflow KindFlow WorkflowKind = "flow" // Orchestrates modules KindFragment WorkflowKind = "fragment" // Reusable step collection ) ``` ### StepType ```go theme={null} const ( StepTypeBash StepType = "bash" StepTypeFunction StepType = "function" StepTypeParallel StepType = "parallel-steps" StepTypeForeach StepType = "foreach" StepTypeRemoteBash StepType = "remote-bash" StepTypeHTTP StepType = "http" StepTypeLLM StepType = "llm" StepTypeFragmentStep StepType = "fragment-step" ) ``` ### RunnerType ```go theme={null} const ( RunnerTypeHost RunnerType = "host" // Local execution RunnerTypeDocker RunnerType = "docker" // Docker container RunnerTypeSSH RunnerType = "ssh" // Remote SSH ) ``` ### TriggerType ```go theme={null} const ( TriggerCron TriggerType = "cron" // Cron schedule TriggerEvent TriggerType = "event" // Event-driven TriggerWatch TriggerType = "watch" // File system watch TriggerManual TriggerType = "manual" // CLI execution ) ``` ## Step Executors | Executor | Description | Runner | | ---------------------- | ------------------------- | --------------- | | `BashExecutor` | Execute shell commands | Host/Docker/SSH | | `FunctionExecutor` | Execute utility functions | Goja runtime | | `ParallelExecutor` | Concurrent step execution | Multiple | | `ForeachExecutor` | Iterate with parallelism | Multiple | | `RemoteBashExecutor` | Remote command execution | Docker/SSH | | `HTTPExecutor` | HTTP requests | Built-in | | `LLMExecutor` | LLM API calls | Built-in | | `FragmentStepExecutor` | Inline fragment execution | Dispatcher | ## Template Engine The template engine provides `{{variable}}` interpolation: ``` +-------------------------------------------------------------+ | Template Engine | | | | Input: "nuclei -l {{Output}}/urls.txt -o {{Output}}/out" | | | | | v | | Context: { Output: "/workspaces/example_com" } | | | | | v | | Output: "nuclei -l /workspaces/example_com/urls.txt ..." | | | | Features: | | - Standard templates: {{variable}} | | - Secondary (foreach): [[variable]] | | - Generators: $rand(16), $uuid() | | - Sharded caching for high concurrency | | - Pre-compiled templates for workflows | | | +-------------------------------------------------------------+ ``` ## Function Runtime Functions are executed via the Goja JavaScript runtime with VM pooling: ``` +-------------------------------------------------------------+ | Goja Runtime Pool | | | | +---------+ +---------+ +---------+ +---------+ | | | VM1 | | VM2 | | VM3 | | VM4 | | | | (idle) | | (busy) | | (idle) | | (busy) | | | +---------+ +---------+ +---------+ +---------+ | | | | Features: | | - Pool size based on CPU cores | | - No global mutex for parallel execution | | - Lazy variable loading for conditions | | - Context isolation per execution | | | +-------------------------------------------------------------+ ``` ## Database Schema ``` +-----------------+ +-----------------+ | Workspaces | | Assets | +-----------------+ +-----------------+ | id | | id | | name | | workspace |--+ | target | | asset_value | | | created_at | | asset_type | | | total_subs | | url | | | total_urls | | status_code | | | ... | | created_at | | +-----------------+ | updated_at | | +-----------------+ | +-----------------+ | | Vulnerabilities | | +-----------------+ | | id | | | workspace |--------------------------+ | asset_value | | vuln_info | | severity | | template_id | | created_at | +-----------------+ ``` ## Scheduler The scheduler manages automated workflow triggers: ``` +-------------------------------------------------------------+ | Scheduler | | | | Trigger Types: | | +------------------------------------------------------+ | | | Cron | Time-based scheduling (cron syntax) | | | | Event | System events (assets.new, etc.) | | | | Watch | File system changes (fsnotify) | | | | Manual | CLI invocation | | | +------------------------------------------------------+ | | | | Event Topics: | | - assets.new New asset discovered | | - vulnerabilities.new New vulnerability found | | - webhook.received External webhook received | | - db.change Database change event | | - watch.files File system change event | | | +-------------------------------------------------------------+ ``` ## Decision Routing Steps support conditional branching using switch/case syntax: ```yaml theme={null} decision: switch: "{{variable}}" cases: "value1": { goto: step-a } "value2": { goto: step-b } default: { goto: fallback } ``` Use `goto: _end` to terminate the workflow. ## Plugin Registry Pattern Step executors are registered in a plugin registry for extensibility: ```go theme={null} type StepExecutor interface { Name() string StepTypes() []core.StepType Execute(ctx context.Context, step *core.Step, execCtx *core.ExecutionContext) (*core.StepResult, error) CanHandle(stepType core.StepType) bool } // Registration dispatcher.RegisterExecutor(NewBashExecutor()) dispatcher.RegisterExecutor(NewFunctionExecutor()) dispatcher.RegisterExecutor(NewFragmentStepExecutor()) // ... ``` ## Configuration Configuration is loaded from `~/osmedeus-base/osm-settings.yaml`: ```yaml theme={null} general: base_folder: ~/osmedeus-base workspaces_path: ~/workspaces-osmedeus binaries_path: ~/osmedeus-base/external-binaries database: type: sqlite # or postgres path: ~/osmedeus-base/osmedeus.db notification: telegram: bot_token: "" chat_id: "" webhooks: [] cdn: enabled: false provider: s3 bucket: "" ``` ## Adding New Features ### New Step Type 1. Add constant in `internal/core/types.go`: ```go theme={null} StepTypeCustom StepType = "custom" ``` 2. Create executor implementing `StepExecutor` in `internal/executor/`: ```go theme={null} type CustomExecutor struct {} func (e *CustomExecutor) Name() string { return "custom" } func (e *CustomExecutor) StepTypes() []core.StepType { return []core.StepType{core.StepTypeCustom} } func (e *CustomExecutor) Execute(...) (*core.StepResult, error) { ... } ``` 3. Register in `dispatcher.go`: ```go theme={null} dispatcher.RegisterExecutor(NewCustomExecutor()) ``` ### New Runner 1. Implement Runner interface in `internal/runner/`: ```go theme={null} type CustomRunner struct {} func (r *CustomRunner) Execute(ctx context.Context, cmd string) (string, error) { ... } func (r *CustomRunner) Close() error { ... } ``` 2. Add type constant and register in runner factory. ### New Utility Function 1. Add Go implementation in `internal/functions/`: ```go theme={null} func (vf *vmFunc) customFunc(call goja.FunctionCall) goja.Value { ... } ``` 2. Add constant in `constants.go`: ```go theme={null} FnCustomFunc = "custom_func" ``` 3. Register in `goja_runtime.go`: ```go theme={null} vm.Set(FnCustomFunc, vf.customFunc) ``` ### New CLI Command 1. Create in `pkg/cli/`: ```go theme={null} var customCmd = &cobra.Command{ Use: "custom", Short: "Custom command", RunE: func(cmd *cobra.Command, args []string) error { ... }, } ``` 2. Add to `rootCmd` in `init()`. ### New API Endpoint 1. Add handler in `pkg/server/handlers/`: ```go theme={null} func CustomHandler(cfg *config.Config) fiber.Handler { ... } ``` 2. Register route in `server.go`. 3. Document in `docs/api/`. # Runners Source: https://docs.osmedeus.org/architecture/runners Execution environments for commands and steps Runners execute commands in different environments. ## Runner Types | Runner | Description | Use Case | | -------- | ------------------------ | -------------------------------------- | | `host` | Local machine execution | Default, fast, no isolation | | `docker` | Container execution | Isolated, reproducible, tool packaging | | `ssh` | Remote machine execution | Distributed scanning, remote resources | ## Host Runner Executes commands on the local machine using the shell. ### Configuration ```yaml theme={null} kind: module name: local-scan runner: host # Default, can be omitted steps: - name: scan type: bash command: nmap -sV {{target}} ``` ### Characteristics * Uses `sh -c` for command execution * Inherits environment from Osmedeus process * No isolation between steps * Fastest execution ## Docker Runner Executes commands inside Docker containers. ### Module-Level Configuration ```yaml theme={null} kind: module name: docker-scan runner: docker runner_config: image: projectdiscovery/nuclei:latest volumes: - "{{Output}}:/output" environment: - "API_KEY={{api_key}}" persistent: false # Default: ephemeral containers steps: - name: scan type: bash command: nuclei -u {{target}} -o /output/nuclei.txt ``` ### Runner Config Options | Option | Type | Description | | ------------- | --------- | ------------------------------------ | | `image` | string | Docker image (required) | | `volumes` | \[]string | Volume mounts (`host:container`) | | `environment` | \[]string | Environment variables | | `persistent` | bool | Keep container running between steps | | `network` | string | Docker network name | | `extra_args` | \[]string | Additional docker run arguments | ### Execution Modes **Ephemeral (default)**: Each step runs `docker run --rm` ```yaml theme={null} runner_config: image: alpine:latest persistent: false # New container per step ``` **Persistent**: Container stays running, steps use `docker exec` ```yaml theme={null} runner_config: image: alpine:latest persistent: true # Reuse container ``` ### Per-Step Docker (remote-bash) Use Docker for specific steps without module-level runner: ```yaml theme={null} kind: module name: hybrid runner: host steps: - name: local-step type: bash command: echo "Running locally" - name: docker-step type: remote-bash step_runner: docker step_runner_config: image: alpine:latest volumes: - "{{Output}}:/output" command: cat /etc/os-release ``` ## SSH Runner Executes commands on remote machines via SSH. ### Module-Level Configuration ```yaml theme={null} kind: module name: remote-scan runner: ssh runner_config: host: scanner.example.com port: 22 user: scanner key_file: ~/.ssh/scanner_key # OR password: secret # Not recommended steps: - name: scan type: bash command: nmap -sV {{target}} ``` ### Runner Config Options | Option | Type | Description | | ------------- | ------ | -------------------------- | | `host` | string | SSH hostname (required) | | `port` | int | SSH port (default: 22) | | `user` | string | SSH username (required) | | `key_file` | string | Path to private key | | `password` | string | SSH password (less secure) | | `known_hosts` | string | Path to known\_hosts file | ### Per-Step SSH (remote-bash) Use SSH for specific steps: ```yaml theme={null} kind: module name: hybrid runner: host steps: - name: local-prep type: bash command: echo {{target}} > /tmp/target.txt - name: remote-scan type: remote-bash step_runner: ssh step_runner_config: host: "{{ssh_host}}" port: 22 user: "{{ssh_user}}" key_file: ~/.ssh/id_rsa command: nmap -sV {{target}} ``` ### File Transfer Copy files from remote to local: ```yaml theme={null} - name: remote-scan type: remote-bash step_runner: ssh step_runner_config: host: scanner.example.com user: scanner key_file: ~/.ssh/key command: nmap -sV {{target}} -oN /tmp/result.txt step_remote_file: /tmp/result.txt host_output_file: "{{Output}}/nmap-result.txt" ``` ## Runner Interface All runners implement this interface: ```go theme={null} type Runner interface { Execute(ctx context.Context, command string) (*CommandResult, error) Setup(ctx context.Context) error Cleanup(ctx context.Context) error Type() core.RunnerType IsRemote() bool } type CommandResult struct { Output string ExitCode int Error error } ``` ## Lifecycle ``` 1. Setup() - Initialize runner (connect SSH, start container) 2. Execute() - Run commands (called per step) 3. Cleanup() - Tear down (disconnect, remove container) ``` ## Choosing a Runner | Scenario | Recommended Runner | | ------------------------ | ------------------------- | | Simple local scans | `host` | | Tool isolation | `docker` | | Reproducible builds | `docker` | | Remote server with tools | `ssh` | | Distributed scanning | `ssh` or distributed mode | | Mixed environments | `remote-bash` per step | ## Best Practices ### Docker 1. **Use specific image tags** ```yaml theme={null} image: projectdiscovery/nuclei:v2.9.0 # Good image: projectdiscovery/nuclei:latest # Less predictable ``` 2. **Mount only needed volumes** ```yaml theme={null} volumes: - "{{Output}}:/output:rw" - "{{Data}}/templates:/templates:ro" ``` 3. **Use persistent mode for many steps** ```yaml theme={null} runner_config: persistent: true # Faster for multi-step workflows ``` ### SSH 1. **Use key authentication** ```yaml theme={null} key_file: ~/.ssh/scanner_key # Avoid: password: secret ``` 2. **Parameterize host details** ```yaml theme={null} runner_config: host: "{{ssh_host}}" user: "{{ssh_user}}" ``` 3. **Check remote tool availability** ```yaml theme={null} - name: check-tools type: bash command: which nmap nuclei httpx ``` ### remote-bash 1. **Use for hybrid workflows** * Local file preparation * Remote heavy scanning * Local result processing 2. **Transfer results back** ```yaml theme={null} step_remote_file: /remote/output.txt host_output_file: "{{Output}}/output.txt" ``` ## Next Steps * [Step Types](../workflows/step-types) - Using remote-bash * [Deployment](../getting-started/deployment) - Distributed mode * [Extending Runners](../extending/runners) - Custom runners # Template Engine Architecture Source: https://docs.osmedeus.org/architecture/templates Deep dive into the Osmedeus template engine implementation # Template Engine Architecture The template engine provides `{{variable}}` interpolation for workflows. It uses the pongo2 template library (Django-compatible) with custom optimizations for high-concurrency scenarios. ## Template Syntax ### Standard Variables Standard template variables use `{{variable}}` syntax: ```yaml theme={null} command: "nuclei -l {{Output}}/urls.txt -o {{Output}}/nuclei.json" ``` ### Secondary Variables (Foreach) Foreach loops use `[[variable]]` syntax to avoid conflicts: ```yaml theme={null} step: type: bash command: "httpx -u [[subdomain]] >> {{Output}}/httpx.txt" ``` ### Generator Functions Generator functions provide dynamic values: ```yaml theme={null} exports: random_id: "$rand(16)" unique_id: "$uuid()" ``` ## Engine Types The template system provides two engine implementations: | Engine | Description | Use Case | | --------------- | ------------------------------- | ---------------- | | `Engine` | Standard single-threaded engine | Simple workflows | | `ShardedEngine` | High-performance sharded cache | High concurrency | ## TemplateEngine Interface ```go theme={null} type TemplateEngine interface { // Render renders a template string with the given context Render(template string, ctx map[string]any) (string, error) // RenderMap renders all template values in a map RenderMap(m map[string]string, ctx map[string]any) (map[string]string, error) // RenderSlice renders all template values in a slice RenderSlice(s []string, ctx map[string]any) ([]string, error) // RenderSecondary renders templates using [[ ]] delimiters RenderSecondary(template string, ctx map[string]any) (string, error) // HasSecondaryVariable checks if template contains [[ ]] HasSecondaryVariable(template string) bool // ExecuteGenerator executes a generator function expression ExecuteGenerator(expr string) (string, error) // RegisterGenerator registers a custom generator function RegisterGenerator(name string, fn GeneratorFunc) } ``` ## BatchRenderer Interface For high-throughput scenarios, the `BatchRenderer` interface extends `TemplateEngine`: ```go theme={null} type BatchRenderer interface { TemplateEngine // RenderBatch renders multiple templates in a single operation // Reduces lock contention by grouping templates RenderBatch(requests []RenderRequest, ctx map[string]any) (map[string]string, error) } type RenderRequest struct { Key string // Identifier (e.g., field name) Template string // Template string to render } ``` ## ShardedEngine Architecture The `ShardedEngine` distributes templates across multiple shards to reduce lock contention: ``` +-------------------------------------------------------------+ | ShardedEngine | | | | Template Hash (FNV-1a) | | | | | v | | +-----------------------------------------------------+ | | | Shard Selection: hash & shardMask | | | +-----------------------------------------------------+ | | | | | +--------+--------+--------+--------+ | | v v v v v | | +--------++--------++--------++--------+... | | |Shard 0 ||Shard 1 ||Shard 2 ||Shard 3 | | | | RWMutex|| RWMutex|| RWMutex|| RWMutex| | | | LRU || LRU || LRU || LRU | | | +--------++--------++--------++--------+ | | | +-------------------------------------------------------------+ ``` ### Configuration ```go theme={null} type ShardedEngineConfig struct { ShardCount int // Number of shards (power of 2) ShardCacheSize int // LRU cache size per shard EnablePooling bool // Use pooled context maps } // Defaults const DefaultShardCount = 16 const DefaultShardCacheSize = 64 ``` ### Performance Features 1. **Sharded Caching**: Templates are distributed across shards using FNV-1a hash, reducing lock contention. 2. **RWMutex per Shard**: Read-heavy workloads benefit from reader-writer locks. 3. **Quick Path**: Templates without `{{` are returned immediately without processing. 4. **Lock-Free Execution**: Compiled pongo2 templates are thread-safe; execution happens outside locks. 5. **Context Pooling**: Reusable context maps reduce GC pressure. ### Render Flow ```go theme={null} func (e *ShardedEngine) Render(template string, ctx map[string]any) (string, error) { // Quick path: no template variables if !strings.Contains(template, "{{") { return template, nil } // Get shard based on template hash shard := e.getShard(template) // Try cache with read lock shard.mu.RLock() tpl, ok := shard.cache.Get(template) shard.mu.RUnlock() if !ok { // Cache miss - parse and cache shard.mu.Lock() // Double-check after write lock tpl, ok = shard.cache.Get(template) if !ok { tpl, err = pongo2.FromString(template) if err != nil { shard.mu.Unlock() return "", err } shard.cache.Add(template, tpl) } shard.mu.Unlock() } // Execute outside lock (pongo2 is thread-safe) return tpl.Execute(pongo2.Context(ctx)) } ``` ## Precompiled Templates The `PrecompiledRegistry` stores pre-compiled templates for workflows: ```go theme={null} type PrecompiledRegistry struct { mu sync.RWMutex workflows map[string]*WorkflowTemplates } type WorkflowTemplates struct { // Key format: "stepName:fieldName" Templates map[string]*pongo2.Template } ``` ### Precompiler Interface ```go theme={null} type Precompiler interface { // PrecompileWorkflow scans and pre-compiles all templates PrecompileWorkflow(workflowName string, templates map[string]string) error // GetPrecompiled retrieves a pre-compiled template GetPrecompiled(workflowName, key string) any // ClearPrecompiled removes pre-compiled templates ClearPrecompiled(workflowName string) } ``` ### Usage ```go theme={null} // At workflow load time registry := NewPrecompiledRegistry() templates := map[string]string{ "scan:command": "nuclei -l {{Output}}/urls.txt", "scan:exports:result": "{{Output}}/nuclei.json", } registry.PrecompileWorkflow("my-workflow", templates) // At execution time if tpl := registry.GetPrecompiled("my-workflow", "scan:command"); tpl != nil { // Use pre-compiled template } ``` ## Context Pooling Context maps are pooled to reduce memory allocations: ```go theme={null} var contextPool = sync.Pool{ New: func() interface{} { return make(map[string]any, 32) }, } func GetContext() map[string]any { return contextPool.Get().(map[string]any) } func PutContext(ctx map[string]any) { clear(ctx) contextPool.Put(ctx) } ``` ### Normalized Boolean Handling Boolean values are normalized for template compatibility: ```go theme={null} func NormalizeBoolsToPooled(ctx map[string]any) map[string]any { pooled := GetContext() for k, v := range ctx { switch val := v.(type) { case bool: pooled[k] = val case string: if val == "true" { pooled[k] = true } else if val == "false" { pooled[k] = false } else { pooled[k] = v } default: pooled[k] = v } } return pooled } ``` ## Generator Functions Generator functions produce dynamic values: ```go theme={null} type GeneratorFunc func(args ...string) (string, error) // Built-in generators var builtinGenerators = map[string]GeneratorFunc{ "rand": func(args ...string) (string, error) { length := 8 if len(args) > 0 { length, _ = strconv.Atoi(args[0]) } return generateRandomString(length), nil }, "uuid": func(args ...string) (string, error) { return uuid.New().String(), nil }, "timestamp": func(args ...string) (string, error) { return strconv.FormatInt(time.Now().Unix(), 10), nil }, } ``` ### Usage ```yaml theme={null} exports: random_id: "$rand(16)" unique_id: "$uuid()" time: "$timestamp()" ``` ## Batch Rendering For high-throughput scenarios, batch rendering reduces lock acquisitions: ```go theme={null} func (e *ShardedEngine) RenderBatch(requests []RenderRequest, ctx map[string]any) (map[string]string, error) { results := make(map[string]string, len(requests)) // Prepare context once processedCtx := NormalizeBoolsToPooled(ctx) defer PutContext(processedCtx) // Group by shard shardGroups := make(map[uint32][]RenderRequest) for _, req := range requests { if !strings.Contains(req.Template, "{{") { results[req.Key] = req.Template continue } idx := e.getShardIndex(req.Template) shardGroups[idx] = append(shardGroups[idx], req) } // Process each shard group for idx, reqs := range shardGroups { shard := e.shards[idx] if err := e.renderShardBatch(shard, reqs, processedCtx, results); err != nil { return nil, err } } return results, nil } ``` ## Secondary Template Rendering Foreach loops use `[[variable]]` to avoid conflicts with pre-rendered `{{variables}}`: ```go theme={null} func (e *ShardedEngine) RenderSecondary(template string, ctx map[string]any) (string, error) { if !strings.Contains(template, "[[") { return template, nil } // Convert [[ ]] to {{ }} for pongo2 converted := strings.ReplaceAll(template, "[[", "{{") converted = strings.ReplaceAll(converted, "]]", "}}") return e.Render(converted, ctx) } ``` ## Performance Benchmarks Typical performance characteristics: | Operation | Single-threaded | 10 Concurrent | 100 Concurrent | | ----------------- | --------------- | ------------- | -------------- | | Simple render | 500ns | 600ns | 800ns | | Cached render | 200ns | 250ns | 400ns | | Batch render (10) | 1.5µs | 2µs | 3µs | | Pre-compiled | 150ns | 200ns | 350ns | ## Best Practices 1. **Use pre-compilation** for frequently executed workflows 2. **Enable pooling** for high-concurrency scenarios 3. **Use batch rendering** when rendering multiple templates with the same context 4. **Avoid unnecessary templates** - only use `{{}}` when variable substitution is needed 5. **Use secondary syntax** `[[]]` for foreach loop variables to avoid double-rendering issues # Workflow Architecture Source: https://docs.osmedeus.org/architecture/workflows Deep dive into the Osmedeus workflow system including fragments and linting # Workflow Architecture Workflows are the core abstraction in Osmedeus, defining automated security tasks through YAML configuration. This document covers the workflow system architecture, including fragments and the linting system. ## Workflow Kinds Osmedeus supports three workflow kinds: | Kind | Purpose | Contains | | ---------- | ------------------------ | ------------- | | `module` | Single execution unit | Steps array | | `flow` | Orchestrate modules | Modules array | | `fragment` | Reusable step collection | Steps array | ``` +-------------------------------------------------------------+ | Flow | | +-------------+ +-------------+ +-------------+ | | | Module A | | Module B | | Module C | | | | +-------+ | | +-------+ | | +-------+ | | | | | Step | | | | Step | | | |Fragment| | | | | | Step | | | | Step | | | | Step | | | | | |Fragment| | | +-------+ | | | Step | | | | | +-------+ | | | | +-------+ | | | +-------------+ +-------------+ +-------------+ | +-------------------------------------------------------------+ ``` ## Workflow Structure ### Workflow Type ```go theme={null} type Workflow struct { Kind WorkflowKind // module, flow, fragment Name string // Unique identifier Description string // Human-readable description Tags TagList // Comma-separated tags Params []Param // Input parameters Triggers []Trigger // Automated triggers Dependencies *Dependencies // External tool requirements Reports []Report // Output reports // Execution preferences Preferences *Preferences // Optional execution settings // Runner configuration (module-kind only) Runner RunnerType // host, docker, ssh RunnerConfig *RunnerConfig // Runner-specific config // Module-specific fields Steps []Step // Execution steps Includes []FragmentInclude // Fragment includes // Flow-specific fields Modules []ModuleRef // Module references // Internal metadata FilePath string // Source file path Checksum string // Content checksum } ``` ### Step Type ```go theme={null} type Step struct { Name string // Unique step identifier Type StepType // Step type DependsOn []string // Step dependencies StepRunner RunnerType // Per-step runner override PreCondition string // Skip if false Log string // Log file path Timeout StepTimeout // Execution timeout // Bash step fields Command string Commands []string ParallelCommands []string StdFile string // Stdout capture file // Structured argument fields SpeedArgs string ConfigArgs string InputArgs string OutputArgs string // Function step fields Function string Functions []string ParallelFunctions []string // Parallel step fields ParallelSteps []Step // Foreach step fields Input string Variable string Threads StepThreads Step *Step // Remote-bash step fields StepRunnerConfig *StepRunnerConfig StepRemoteFile string HostOutputFile string // HTTP step fields URL string Method string Headers map[string]string RequestBody string // LLM step fields Messages []LLMMessage Tools []LLMTool LLMConfig *LLMStepConfig IsEmbedding bool EmbeddingInput []string // Fragment-step fields FragmentName string // Fragment to execute Override map[string]string // Override parameters // Common fields Exports map[string]string OnSuccess []Action OnError []Action Decision *DecisionConfig } ``` ## Workflow Inheritance Workflows support inheritance through the `extends` field, allowing child workflows to inherit and override parent configurations. ### Inheritance Architecture ``` +-------------------------------------------------------------+ | InheritanceResolver | | | | Child Workflow | | +---------------------------------------------------------+ | | extends: parent-workflow | | | override: { params: ..., steps: ... } | | +---------------------------------------------------------+ | | | | v | | +---------------------------------------------------------+ | | 1. Check for circular dependency | | | 2. Load parent workflow | | | 3. Recursively resolve parent's inheritance | | | 4. Validate kind compatibility | | | 5. Merge parent -> child with overrides | | +---------------------------------------------------------+ | | | | v | | Merged Workflow | | +---------------------------------------------------------+ | | All parent fields + child overrides | | | ResolvedFrom: "parent-workflow" | | +---------------------------------------------------------+ +-------------------------------------------------------------+ ``` ### InheritanceResolver Type ```go theme={null} type InheritanceResolver struct { loader *Loader resolving map[string]bool // Track workflows being resolved (circular detection) childPath string // Directory of current child (for relative resolution) } ``` ### Resolution Process 1. **Circular Detection**: Track workflows being resolved to detect circular inheritance 2. **Parent Loading**: Load parent by name (same directory) or path (relative/absolute) 3. **Recursive Resolution**: If parent also extends, resolve recursively 4. **Kind Validation**: Child and parent must have matching `kind` (module/flow) 5. **Merge**: Apply child's direct fields and override section ### Override Modes ```go theme={null} const ( OverrideModeReplace OverrideMode = "replace" // Replace parent items entirely OverrideModePrepend OverrideMode = "prepend" // Add child items before parent OverrideModeAppend OverrideMode = "append" // Add child items after parent (default) OverrideModeMerge OverrideMode = "merge" // Match by name, replace/remove/append ) ``` ### WorkflowOverride Type ```go theme={null} type WorkflowOverride struct { Params map[string]*ParamOverride // Override parameter properties Steps *StepsOverride // Steps override (modules only) Modules *ModulesOverride // Modules override (flows only) Triggers []Trigger // Replace triggers entirely Dependencies *Dependencies // Merge with parent dependencies Preferences *Preferences // Child overrides parent RunnerConfig *RunnerConfig // Child overrides parent Runner *RunnerType // Override runner type } ``` ### StepsOverride Type ```go theme={null} type StepsOverride struct { Mode OverrideMode // replace, prepend, append, merge Steps []Step // Steps to add/match Remove []string // Step names to remove (merge mode) Replace []Step // Steps to replace by name (merge mode) } ``` ### Merge Priority ``` Priority: Child direct fields > Child override > Parent 1. Start with clone of parent workflow 2. Apply child's direct fields (name, description, tags) 3. Apply override section by mode 4. Clear extends field to prevent re-resolution ``` ### Parent Resolution Order 1. Same directory as child (name + .yaml/.yml) 2. Relative path from child's directory 3. Workflows directory search by name 4. Absolute path ## Fragments Fragments are reusable step collections that can be embedded in modules. ### Fragment Definition ```yaml theme={null} kind: fragment name: notification-fragment description: Common notification steps params: - name: channel type: string default: "#security-alerts" steps: - name: notify type: bash command: notify-send "{{channel}}" "{{message}}" ``` ### Fragment Include Modules can include fragments using the `includes` field: ```yaml theme={null} kind: module name: subdomain-enum includes: - name: notification-fragment as: notify params: channel: "#recon" steps: - name: run-amass type: bash command: amass enum -d {{target}} - name: send-notification type: fragment fragment_name: notify override: message: "Subdomain enumeration complete" ``` ### FragmentInclude Type ```go theme={null} type FragmentInclude struct { Name string // Fragment workflow name As string // Local alias Params map[string]string // Parameter overrides } ``` ### Fragment Resolution 1. **Loading**: Fragments are loaded during workflow parsing 2. **Validation**: Fragment kind must be `fragment` 3. **Parameter Binding**: Include params merged with fragment defaults 4. **Step Expansion**: Fragment steps are embedded at execution time ### Fragment Step Type ```go theme={null} type Step struct { // ... other fields Type StepType // "fragment" FragmentName string // Alias from includes Override map[string]string // Runtime parameter overrides } ``` ### Fragment Execution When a fragment step is executed: 1. Resolve fragment from includes by alias 2. Merge override params with include params 3. Execute fragment steps in sequence 4. Return combined results ## Linting System The workflow linter validates YAML workflows for correctness and best practices. ### Linting Architecture ``` +-------------------------------------------------------------+ | Linter | | | | YAML Source | | +---------------------------------------------------------+ | | kind: module | | | name: my-workflow | | | steps: ... | | +---------------------------------------------------------+ | | | | v | | +---------------------------------------------------------+ | | WorkflowAST | | | - Workflow: *core.Workflow | | | - Source: []byte | | | - Root: ast.Node | | | - NodeMap: map[string]ast.Node | | +---------------------------------------------------------+ | | | | v | | +---------------------------------------------------------+ | | Rules | | | +------------------+ +------------------+ | | | | MissingRequired | | DuplicateStepName| | | | +------------------+ +------------------+ | | | +------------------+ +------------------+ | | | | EmptyStep | | UnusedVariable | | | | +------------------+ +------------------+ | | | +------------------+ +------------------+ | | | | InvalidGoto | |InvalidDependsOn | | | | +------------------+ +------------------+ | | | +------------------+ +------------------+ | | | |CircularDependency| |UndefinedVariable | | | | +------------------+ +------------------+ | | +---------------------------------------------------------+ | | | | v | | +---------------------------------------------------------+ | | LintResult | | | - Issues: []LintIssue | | | - Errors: int | | | - Warnings: int | | | - Infos: int | | +---------------------------------------------------------+ +-------------------------------------------------------------+ ``` ### Built-in Rules | Rule | Severity | Description | | ------------------------ | -------- | ------------------------------------------ | | `missing-required-field` | warning | Required fields (name, kind, type) missing | | `duplicate-step-name` | warning | Multiple steps with same name | | `empty-step` | warning | Step has no executable content | | `unused-variable` | info | Variable exported but never used | | `undefined-variable` | warning | Variable referenced but not defined | | `invalid-goto` | warning | Decision goto references non-existent step | | `invalid-depends-on` | warning | depends\_on references non-existent step | | `circular-dependency` | warning | Circular step dependencies detected | ### LinterRule Interface ```go theme={null} type LinterRule interface { Name() string // Unique identifier Description() string // Human-readable description Severity() Severity // Default severity Check(ast *WorkflowAST) []LintIssue } ``` ### LintIssue Type ```go theme={null} type LintIssue struct { Rule string // Rule name Severity Severity // Issue severity Message string // Human-readable description Suggestion string // Fix suggestion Line int // 1-based line number Column int // 1-based column number Field string // YAML path (e.g., "steps[0].bash") } ``` ### Running the Linter #### CLI Usage ```bash theme={null} # Validate by name osmedeus workflow validate subdomain-enum # Validate file osmedeus workflow lint ./my-workflow.yaml # Validate folder osmedeus workflow validate /path/to/workflows/ # CI mode osmedeus workflow lint . --check --format json ``` #### Output Formats **Pretty (default)**: ``` workflows/test.yaml:15:12 warning undefined-variable Variable 'unknown_var' is not defined Suggestion: Check that the variable is defined in params or a previous step's exports ``` **JSON**: ```json theme={null} { "file_path": "workflows/test.yaml", "issues": [ { "rule": "undefined-variable", "severity": "warning", "message": "Variable 'unknown_var' is not defined", "line": 15, "column": 12, "field": "steps[2].command" } ] } ``` **GitHub Actions**: ``` ::warning file=workflows/test.yaml,line=15,col=12::undefined-variable: Variable 'unknown_var' is not defined ``` ### Disabling Rules ```bash theme={null} # Disable specific rules osmedeus workflow lint . --disable unused-variable,empty-step ``` ### Custom Rules Implement the `LinterRule` interface: ```go theme={null} type MyCustomRule struct{} func (r *MyCustomRule) Name() string { return "my-custom-rule" } func (r *MyCustomRule) Description() string { return "Description of what this rule checks" } func (r *MyCustomRule) Severity() Severity { return SeverityWarning } func (r *MyCustomRule) Check(wast *WorkflowAST) []LintIssue { var issues []LintIssue w := wast.Workflow // Implement your validation logic for i, step := range w.Steps { if /* condition */ { line, col := wast.FindStepPosition(step.Name) issues = append(issues, LintIssue{ Rule: r.Name(), Severity: r.Severity(), Message: "Issue description", Suggestion: "How to fix", Line: line, Column: col, Field: fmt.Sprintf("steps[%d]", i), }) } } return issues } // Register the rule linter := linter.NewDefaultLinter() linter.RegisterRule(&MyCustomRule{}) ``` ## Decision Routing Steps support conditional branching: ```yaml theme={null} decision: switch: "{{status}}" cases: "critical": { goto: alert-step } "high": { goto: process-high } "none": { goto: _end } default: { goto: continue-step } ``` ### DecisionConfig Type ```go theme={null} type DecisionConfig struct { Switch string // Variable to evaluate Cases map[string]DecisionCase // Case mappings Default *DecisionCase // Default case } type DecisionCase struct { Goto string // Target step name or "_end" } ``` ## Workflow Execution Context ```go theme={null} type ExecutionContext struct { RunID string Target string Workspace string Output string Variables map[string]interface{} Exports map[string]interface{} StepResults map[string]*StepResult Config *config.Config Runner runner.Runner TemplateEngine template.TemplateEngine Runtime *functions.GojaRuntime } ``` ## Best Practices ### Workflow Design 1. **Use fragments** for reusable step collections 2. **Keep modules focused** - Single responsibility 3. **Use flows** to orchestrate complex pipelines 4. **Leverage decision routing** for adaptive workflows 5. **Define dependencies** with `depends_on` for DAG execution ### Performance 1. **Use parallel\_commands** for independent commands 2. **Use parallel\_functions** for independent function calls 3. **Set appropriate timeouts** to prevent hanging 4. **Use foreach** with appropriate thread counts ### Validation 1. **Run linter** before deploying workflows 2. **Use CI integration** with `--check --format github` 3. **Enable all rules** during development 4. **Use type annotations** in params for validation ### Error Handling 1. **Use on\_error handlers** for graceful failures 2. **Use pre\_condition** to skip steps safely 3. **Export meaningful error information** 4. **Log appropriately** for debugging # Cloud Cheatsheet Source: https://docs.osmedeus.org/cloud/cheatsheet Quick cheatsheet for cloud setup, workflow mode, custom commands, and infrastructure management ## First-Time Setup ```bash theme={null} # 0. Enable cloud feature osmedeus config set cloud.enabled true # 1. Credentials (pick one provider) osmedeus cloud config set providers.aws.access_key_id osmedeus cloud config set providers.aws.secret_access_key osmedeus cloud config set providers.aws.region ap-southeast-1 osmedeus cloud config set defaults.provider aws # 2. SSH osmedeus cloud config set ssh.private_key_path ~/.ssh/id_rsa osmedeus cloud config set ssh.public_key_path ~/.ssh/id_rsa.pub # 3. Clean the setup scripts first, then add worker setup osmedeus cloud config set setup.commands.clear "" osmedeus cloud config set setup.commands.add "curl -fsSL https://www.osmedeus.org/install.sh | bash" osmedeus cloud config set setup.commands.add "osmedeus install base --preset" # 4. Cost limits (recommended) osmedeus cloud config set limits.max_hourly_spend 1.00 osmedeus cloud config set limits.max_total_spend 10.00 ``` ## Workflow Mode ```bash theme={null} osmedeus cloud run -f fast -t example.com # Single target osmedeus cloud run -f fast -T targets.txt --instances 5 # Distributed osmedeus cloud run -f fast -t example.com --sync-back # Sync results osmedeus cloud run -f fast -t example.com --auto-destroy # Auto cleanup osmedeus cloud run -f fast -t example.com --sync-back --auto-destroy # Full lifecycle osmedeus cloud run -f fast -t example.com --reuse # Reuse infra osmedeus cloud run -m enum-subdomain -t example.com --timeout 30m # Module + timeout ``` ## Custom Command Mode ```bash theme={null} # Run anything on cloud instances osmedeus cloud run --custom-cmd "nmap -sV {{Target}}" -t example.com # Pipeline: multiple commands, sync results osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -o /tmp/osm-custom/live.txt" \ --custom-post-cmd "wc -l /tmp/osm-custom/live.txt" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy # Distribute targets, sync to custom dir osmedeus cloud run \ --custom-cmd "cat {{Target}} | nuclei -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/nuclei.txt" \ --sync-dest "./nuclei-results" \ -T targets.txt --instances 5 ``` ### Variables: `{{Target}}` `{{public_ip}}` `{{private_ip}}` `{{worker_name}}` `{{worker_id}}` `{{infra_id}}` `{{provider}}` `{{ssh_user}}` `{{index}}` ### Rules * Commands run in `/tmp/osm-custom/` on remote * Sequential per worker, parallel across workers * First failure skips remaining cmds + post-cmds * Sync destination: `/-/` ## Infrastructure ```bash theme={null} osmedeus cloud create --provider aws -n 3 # Create osmedeus cloud list # List osmedeus cloud destroy # Destroy one osmedeus cloud destroy all --force # Destroy all osmedeus cloud setup --reuse-with "1.2.3.4" # Setup existing ``` ## Config ```bash theme={null} osmedeus cloud config list # View osmedeus cloud config set # Set osmedeus cloud config set .add # Append to list osmedeus cloud config clean # Reset ``` ## Provider Quick Config **AWS:** ```bash theme={null} osmedeus cloud config set providers.aws.access_key_id ${AWS_ACCESS_KEY_ID} osmedeus cloud config set providers.aws.secret_access_key ${AWS_SECRET_ACCESS_KEY} osmedeus cloud config set providers.aws.region ap-southeast-1 osmedeus cloud config set providers.aws.instance_type t3.medium osmedeus cloud config set providers.aws.use_spot true # 70% cheaper ``` **Hetzner:** ```bash theme={null} osmedeus cloud config set providers.hetzner.token ${HETZNER_API_TOKEN} osmedeus cloud config set providers.hetzner.location fsn1 osmedeus cloud config set providers.hetzner.server_type cx22 ``` **DigitalOcean:** ```bash theme={null} osmedeus cloud config set providers.digitalocean.token ${DO_TOKEN} osmedeus cloud config set providers.digitalocean.region sgp1 osmedeus cloud config set providers.digitalocean.size s-2vcpu-4gb ``` **GCP:** ```bash theme={null} osmedeus cloud config set providers.gcp.project_id ${GCP_PROJECT} osmedeus cloud config set providers.gcp.credentials_file /path/to/sa-key.json osmedeus cloud config set providers.gcp.region us-central1 osmedeus cloud config set providers.gcp.zone us-central1-a osmedeus cloud config set providers.gcp.machine_type n1-standard-2 ``` **Linode:** ```bash theme={null} osmedeus cloud config set providers.linode.token ${LINODE_TOKEN} osmedeus cloud config set providers.linode.region ap-south osmedeus cloud config set providers.linode.type g6-standard-2 ``` **Azure:** ```bash theme={null} osmedeus cloud config set providers.azure.subscription_id ${AZURE_SUB_ID} osmedeus cloud config set providers.azure.tenant_id ${AZURE_TENANT_ID} osmedeus cloud config set providers.azure.client_id ${AZURE_CLIENT_ID} osmedeus cloud config set providers.azure.client_secret ${AZURE_CLIENT_SECRET} osmedeus cloud config set providers.azure.location southeastasia osmedeus cloud config set providers.azure.vm_size Standard_B2s ``` ## Cost Reference | Provider | Instance | vCPU | RAM | \$/hr | | ------------ | ------------- | ---- | ------ | ----- | | Hetzner | cx22 | 2 | 4 GB | 0.007 | | Linode | g6-standard-2 | 2 | 4 GB | 0.018 | | DigitalOcean | s-2vcpu-4gb | 2 | 4 GB | 0.022 | | AWS | t3.medium | 2 | 4 GB | 0.042 | | Azure | Standard\_B2s | 2 | 4 GB | 0.042 | | GCP | n1-standard-2 | 2 | 7.5 GB | 0.095 | 5 x Hetzner cx22 x 2 hours = **$0.07** | 5 x DO s-2vcpu-4gb x 2 hours = **$0.22** ## Troubleshooting ```bash theme={null} osmedeus cloud run -f fast -t example.com --verbose-setup # See setup output osmedeus cloud run -f fast -t example.com --debug # Full debug logs osmedeus cloud list # Check for orphans osmedeus cloud destroy all --force # Emergency cleanup ``` # AWS Provider Guide Source: https://docs.osmedeus.org/cloud/provider-aws Step-by-step guide for running osmedeus cloud on AWS EC2 instances web-ui-vuln web-ui-assets Step-by-step guide for running osmedeus cloud on AWS EC2 instances. ## Prerequisites * An AWS account * An IAM user or role with EC2 permissions * An SSH key pair (local `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`) ### Required IAM Permissions The IAM user needs these permissions (or use the `AmazonEC2FullAccess` managed policy): ``` ec2:RunInstances ec2:TerminateInstances ec2:DescribeInstances ec2:DescribeImages ec2:CreateSecurityGroup ec2:AuthorizeSecurityGroupIngress ec2:DeleteSecurityGroup ec2:DescribeSecurityGroups ec2:ImportKeyPair ec2:DeleteKeyPair ec2:DescribeKeyPairs ec2:CreateTags ``` ### Get Your Credentials 1. Go to **IAM Console** > **Users** > select your user 2. **Security credentials** tab > **Create access key** 3. Save the **Access key ID** and **Secret access key** Or use environment variables if already configured for AWS CLI: ```bash theme={null} export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= ``` ## Configuration ### Minimal Setup ```bash theme={null} # Enable cloud feature osmedeus config set cloud.enabled true # Credentials osmedeus cloud config set providers.aws.access_key_id ${AWS_ACCESS_KEY_ID} osmedeus cloud config set providers.aws.secret_access_key ${AWS_SECRET_ACCESS_KEY} osmedeus cloud config set providers.aws.region ap-southeast-1 osmedeus cloud config set defaults.provider aws # SSH osmedeus cloud config set ssh.private_key_path ~/.ssh/id_rsa osmedeus cloud config set ssh.public_key_path ~/.ssh/id_rsa.pub osmedeus cloud config set ssh.user ubuntu # Clean the setup scripts first osmedeus cloud config set setup.commands.clear "" # Worker setup osmedeus cloud config set setup.commands.add "sudo apt-get update" osmedeus cloud config set setup.commands.add "sudo apt-get install -y -qq curl git tmux unzip jq rsync" osmedeus cloud config set setup.commands.add "curl -fsSL https://www.osmedeus.org/install.sh | bash" osmedeus cloud config set setup.commands.add "osmedeus health" ``` ### Instance Types | Instance | vCPU | RAM | \$/hr (on-demand) | \$/hr (spot, \~70% off) | Best For | | ---------- | ---- | ----- | ----------------- | ----------------------- | ------------------------------- | | t3.medium | 2 | 4 GB | \$0.0416 | \~\$0.012 | Light scans, single targets | | t3.large | 2 | 8 GB | \$0.0832 | \~\$0.025 | General scanning | | t3.xlarge | 4 | 16 GB | \$0.1664 | \~\$0.050 | Heavy scans, large target lists | | t3.2xlarge | 8 | 32 GB | \$0.3328 | \~\$0.100 | Parallel pipelines | ```bash theme={null} # Set instance type osmedeus cloud config set providers.aws.instance_type t3.large ``` ### Spot Instances Spot instances cost 60-80% less than on-demand. They can be interrupted but are fine for security scanning (stateless, can retry). ```bash theme={null} osmedeus cloud config set providers.aws.use_spot true ``` ### Regions Pick a region close to your targets or with the lowest pricing: | Region | Location | Code | | ------------------------ | --------- | ---------------- | | US East (N. Virginia) | US | `us-east-1` | | US West (Oregon) | US | `us-west-2` | | EU (Frankfurt) | Europe | `eu-central-1` | | EU (Ireland) | Europe | `eu-west-1` | | Asia Pacific (Singapore) | Asia | `ap-southeast-1` | | Asia Pacific (Tokyo) | Asia | `ap-northeast-1` | | Asia Pacific (Mumbai) | Asia | `ap-south-1` | | Asia Pacific (Sydney) | Australia | `ap-southeast-2` | ```bash theme={null} osmedeus cloud config set providers.aws.region us-east-1 ``` ### Custom AMI Use a custom AMI with tools pre-installed for faster startup: ```bash theme={null} # Find the default Ubuntu AMI for your region # aws ec2 describe-images --owners 099720109477 --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-*-amd64-*" --query 'sort_by(Images, &CreationDate)[-1].ImageId' # Or use your own pre-built AMI osmedeus cloud config set providers.aws.ami ami-0123456789abcdef0 ``` ### Cost Limits ```bash theme={null} osmedeus cloud config set limits.max_hourly_spend 1.00 osmedeus cloud config set limits.max_total_spend 10.00 osmedeus cloud config set limits.max_instances 10 ``` ## Examples ### Quick Domain Recon ```bash theme={null} osmedeus cloud run -f fast -t example.com --auto-destroy ``` Cost: \~\$0.04 (1 x t3.medium x 1 hour) ### Large-Scale Subdomain Enumeration ```bash theme={null} # targets.txt: one domain per line osmedeus cloud run -f general -T targets.txt --instances 5 --sync-back --auto-destroy ``` Cost: \~\$0.42 (5 x t3.medium x 2 hours) ### Custom Nmap Scan ```bash theme={null} osmedeus cloud run \ --custom-cmd "nmap -sV -sC {{Target}} -oA /tmp/osm-custom/nmap" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy ``` ### Distributed Nuclei Scanning ```bash theme={null} osmedeus cloud run \ --custom-cmd "cat {{Target}} | nuclei -o /tmp/osm-custom/results.txt" \ --sync-path "/tmp/osm-custom/results.txt" \ --sync-dest "./nuclei-aws" \ -T urls.txt --instances 10 --auto-destroy ``` Cost: \~\$0.42 (10 x t3.medium x 1 hour) ### Spot Instance Pipeline ```bash theme={null} # Configure spot osmedeus cloud config set providers.aws.use_spot true osmedeus cloud config set providers.aws.instance_type t3.large # Run a heavy scan for cheap osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -all -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -td -o /tmp/osm-custom/live.txt" \ --custom-cmd "cat /tmp/osm-custom/live.txt | nuclei -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy ``` Cost: \~\$0.025 (1 x t3.large spot x 1 hour) ### Persistent Recon Campaign ```bash theme={null} # Create instances once (saves setup time on subsequent runs) osmedeus cloud create --provider aws -n 3 # Run scans throughout the day osmedeus cloud run -f fast -t target1.com --reuse osmedeus cloud run -f fast -t target2.com --reuse osmedeus cloud run --custom-cmd "nuclei -u target3.com -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/" -t target3.com --reuse # Destroy at end of day osmedeus cloud destroy all --force ``` ### Multi-Region Scanning ```bash theme={null} # Scan US targets from US region osmedeus cloud config set providers.aws.region us-east-1 osmedeus cloud run -f fast -t us-company.com --auto-destroy # Scan APAC targets from Singapore osmedeus cloud config set providers.aws.region ap-southeast-1 osmedeus cloud run -f fast -t apac-company.com --auto-destroy ``` ## Troubleshooting ### "UnauthorizedOperation" Error Your IAM user lacks required permissions. Attach `AmazonEC2FullAccess` policy or the minimal permissions listed above. ### Instances Not Starting ```bash theme={null} # Check with debug output osmedeus cloud run -f fast -t example.com --debug # Common causes: # - Region doesn't have the instance type available # - vCPU limit reached (request limit increase in AWS console) # - Spot capacity unavailable (try on-demand or different region) ``` ### SSH Connection Timeout ```bash theme={null} # Verify security group allows SSH (port 22) # Check with verbose setup osmedeus cloud run -f fast -t example.com --verbose-setup ``` ### Spot Instance Interrupted Spot instances can be reclaimed by AWS. The scan will fail for that worker. Mitigation: * Use `--auto-destroy` to clean up * Re-run the failed targets * Use on-demand instances for critical scans ### Cleaning Up ```bash theme={null} # List all infrastructure osmedeus cloud list # Destroy specific osmedeus cloud destroy # Nuclear option osmedeus cloud destroy all --force # If osmedeus state is out of sync, check AWS console directly: # EC2 Console > Instances > filter by tag "osmedeus" ``` ## Cost Optimization 1. **Use spot instances** for all non-critical scans (`use_spot: true`) 2. **Right-size instances**: t3.medium is enough for most single-target scans 3. **Always use `--auto-destroy`** to prevent forgotten instances 4. **Set cost limits** to catch runaway spending 5. **Use custom AMIs** to reduce setup time (less instance-hours) 6. **Pick the cheapest region** if target geo-location doesn't matter (us-east-1 is usually cheapest) # GCP Provider Guide Source: https://docs.osmedeus.org/cloud/provider-gcp Step-by-step guide for running osmedeus cloud on Google Cloud Platform Compute Engine instances Step-by-step guide for running osmedeus cloud on Google Cloud Platform Compute Engine instances. ## Prerequisites * A GCP account with a project * A service account with Compute Engine permissions * A service account key file (JSON) * An SSH key pair (local `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`) ### Required IAM Permissions The service account needs these roles (or use the `Compute Admin` role): ``` compute.instances.create compute.instances.delete compute.instances.get compute.instances.list compute.instances.setMetadata compute.firewalls.create compute.firewalls.delete compute.firewalls.get compute.networks.get compute.subnetworks.use compute.disks.create compute.images.get compute.images.useReadOnly ``` The simplest approach is to assign the **Compute Admin** (`roles/compute.admin`) role to your service account. ### Create a Service Account and Key 1. Go to **IAM & Admin** > **Service Accounts** > **Create Service Account** 2. Name it `osmedeus-cloud` (or similar) 3. Grant it the **Compute Admin** role 4. Go to the service account > **Keys** > **Add Key** > **Create new key** > **JSON** 5. Save the JSON file (e.g., `~/.gcp/osmedeus-sa.json`) Or via `gcloud` CLI: ```bash theme={null} # Create service account gcloud iam service-accounts create osmedeus-cloud \ --display-name="Osmedeus Cloud Scanner" # Grant Compute Admin role gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member="serviceAccount:osmedeus-cloud@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/compute.admin" # Create and download key file gcloud iam service-accounts keys create ~/.gcp/osmedeus-sa.json \ --iam-account=osmedeus-cloud@YOUR_PROJECT_ID.iam.gserviceaccount.com ``` You can also export the credentials file path as an environment variable: ```bash theme={null} export GCP_PROJECT_ID=your-project-id export GCP_CREDENTIALS_FILE=~/.gcp/osmedeus-sa.json ``` ## Configuration ### Minimal Setup ```bash theme={null} # Enable cloud feature osmedeus config set cloud.enabled true # Credentials osmedeus cloud config set providers.gcp.project_id ${GCP_PROJECT_ID} osmedeus cloud config set providers.gcp.credentials_file ${GCP_CREDENTIALS_FILE} osmedeus cloud config set providers.gcp.region us-central1 osmedeus cloud config set providers.gcp.zone us-central1-a osmedeus cloud config set defaults.provider gcp # SSH osmedeus cloud config set ssh.private_key_path ~/.ssh/id_rsa osmedeus cloud config set ssh.public_key_path ~/.ssh/id_rsa.pub osmedeus cloud config set ssh.user root # Clean the setup scripts first osmedeus cloud config set setup.commands.clear "" # Worker setup osmedeus cloud config set setup.commands.add "sudo apt-get update" osmedeus cloud config set setup.commands.add "sudo apt-get install -y -qq curl git tmux unzip jq rsync" osmedeus cloud config set setup.commands.add "curl -fsSL https://www.osmedeus.org/install.sh | bash" osmedeus cloud config set setup.commands.add "osmedeus install base --preset" ``` ### Machine Types | Machine Type | vCPU | RAM | \$/hr (on-demand) | \$/hr (preemptible, \~80% off) | Best For | | ------------- | ---- | ------ | ----------------- | ------------------------------ | ------------------------------- | | e2-medium | 2 | 4 GB | \$0.0335 | \~\$0.010 | Light scans, single targets | | n1-standard-2 | 2 | 7.5 GB | \$0.0950 | \~\$0.019 | General scanning (default) | | n1-standard-4 | 4 | 15 GB | \$0.1900 | \~\$0.038 | Heavy scans, large target lists | | n2-standard-2 | 2 | 8 GB | \$0.0971 | \~\$0.019 | General scanning (newer gen) | | n2-standard-4 | 4 | 16 GB | \$0.1942 | \~\$0.039 | Parallel pipelines | | c2-standard-4 | 4 | 16 GB | \$0.2088 | \~\$0.042 | CPU-intensive scans | ```bash theme={null} # Set machine type osmedeus cloud config set providers.gcp.machine_type n1-standard-2 ``` ### Preemptible Instances Preemptible VMs cost up to 80% less than on-demand. They last at most 24 hours and can be reclaimed, but are ideal for security scanning workloads. ```bash theme={null} osmedeus cloud config set providers.gcp.use_preemptible true ``` ### Regions and Zones Pick a region close to your targets or with the lowest pricing: | Region | Location | Code | Zone Example | | -------------- | --------- | ---------------------- | ------------------------ | | Iowa | US | `us-central1` | `us-central1-a` | | South Carolina | US | `us-east1` | `us-east1-b` | | Oregon | US | `us-west1` | `us-west1-b` | | Frankfurt | Europe | `europe-west3` | `europe-west3-a` | | London | Europe | `europe-west2` | `europe-west2-a` | | Singapore | Asia | `asia-southeast1` | `asia-southeast1-a` | | Tokyo | Asia | `asia-northeast1` | `asia-northeast1-a` | | Mumbai | Asia | `asia-south1` | `asia-south1-a` | | Sydney | Australia | `australia-southeast1` | `australia-southeast1-a` | ```bash theme={null} osmedeus cloud config set providers.gcp.region us-central1 osmedeus cloud config set providers.gcp.zone us-central1-a ``` > **Note:** The zone must be within the selected region. ### Custom Image Family Use a custom image family with tools pre-installed for faster startup: ```bash theme={null} # Default is ubuntu-2204-lts from the ubuntu-os-cloud project # Use your own custom image family if you have one osmedeus cloud config set providers.gcp.image_family my-osmedeus-image ``` ### Cost Limits ```bash theme={null} osmedeus cloud config set limits.max_hourly_spend 1.00 osmedeus cloud config set limits.max_total_spend 10.00 osmedeus cloud config set limits.max_instances 10 ``` ## Examples ### Quick Domain Recon ```bash theme={null} osmedeus cloud run -f fast -t example.com --auto-destroy ``` Cost: \~\$0.03 (1 x e2-medium x 1 hour) ### Large-Scale Subdomain Enumeration ```bash theme={null} # targets.txt: one domain per line osmedeus cloud run -f general -T targets.txt --instances 5 --sync-back --auto-destroy ``` Cost: \~\$0.48 (5 x n1-standard-2 x 1 hour) ### Custom Nmap Scan ```bash theme={null} osmedeus cloud run \ --custom-cmd "nmap -sV -sC {{Target}} -oA /tmp/osm-custom/nmap" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy ``` ### Distributed Nuclei Scanning ```bash theme={null} osmedeus cloud run \ --custom-cmd "cat {{Target}} | nuclei -o /tmp/osm-custom/results.txt" \ --sync-path "/tmp/osm-custom/results.txt" \ --sync-dest "./nuclei-gcp" \ -T urls.txt --instances 10 --auto-destroy ``` Cost: \~\$0.34 (10 x e2-medium x 1 hour) ### Preemptible Instance Pipeline ```bash theme={null} # Configure preemptible osmedeus cloud config set providers.gcp.use_preemptible true osmedeus cloud config set providers.gcp.machine_type n1-standard-2 # Run a heavy scan for cheap osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -all -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -td -o /tmp/osm-custom/live.txt" \ --custom-cmd "cat /tmp/osm-custom/live.txt | nuclei -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy ``` Cost: \~\$0.019 (1 x n1-standard-2 preemptible x 1 hour) ### Persistent Recon Campaign ```bash theme={null} # Create instances once (saves setup time on subsequent runs) osmedeus cloud create --provider gcp -n 3 # Run scans throughout the day osmedeus cloud run -f fast -t target1.com --reuse osmedeus cloud run -f fast -t target2.com --reuse osmedeus cloud run --custom-cmd "nuclei -u target3.com -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/" -t target3.com --reuse # Destroy at end of day osmedeus cloud destroy all --force ``` ### Multi-Region Scanning ```bash theme={null} # Scan US targets from Iowa osmedeus cloud config set providers.gcp.region us-central1 osmedeus cloud config set providers.gcp.zone us-central1-a osmedeus cloud run -f fast -t us-company.com --auto-destroy # Scan APAC targets from Singapore osmedeus cloud config set providers.gcp.region asia-southeast1 osmedeus cloud config set providers.gcp.zone asia-southeast1-a osmedeus cloud run -f fast -t apac-company.com --auto-destroy ``` ## Troubleshooting ### "Permission denied" or "403 Forbidden" Your service account lacks required permissions. Assign the **Compute Admin** role: ```bash theme={null} gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member="serviceAccount:YOUR_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/compute.admin" ``` ### "Credentials file not found" Make sure the JSON key file path is correct and the file exists: ```bash theme={null} # Check the file exists ls -la ~/.gcp/osmedeus-sa.json # Or set via environment variable export GCP_CREDENTIALS_FILE=/absolute/path/to/key.json osmedeus cloud config set providers.gcp.credentials_file ${GCP_CREDENTIALS_FILE} ``` ### Instances Not Starting ```bash theme={null} # Check with debug output osmedeus cloud run -f fast -t example.com --debug # Common causes: # - Quota exceeded (check Quotas page in Cloud Console) # - Zone doesn't have the machine type available # - Compute Engine API not enabled (enable it in APIs & Services) # - Preemptible capacity unavailable (try a different zone or on-demand) ``` ### "Compute Engine API has not been used" Error Enable the Compute Engine API for your project: ```bash theme={null} gcloud services enable compute.googleapis.com --project=YOUR_PROJECT_ID ``` ### SSH Connection Timeout ```bash theme={null} # Verify firewall rule allows SSH (port 22) gcloud compute firewall-rules list --filter="name~osmedeus" # Check with verbose setup osmedeus cloud run -f fast -t example.com --verbose-setup ``` ### Preemptible Instance Terminated Preemptible VMs are reclaimed after 24 hours or when GCP needs capacity. The scan will fail for that worker. Mitigation: * Use `--auto-destroy` to clean up * Re-run the failed targets * Use on-demand instances for critical or long-running scans ### Cleaning Up ```bash theme={null} # List all infrastructure osmedeus cloud list # Destroy specific osmedeus cloud destroy # Nuclear option osmedeus cloud destroy all --force # If osmedeus state is out of sync, check GCP console directly: # Compute Engine > VM Instances > filter by label "osmedeus" # Or via gcloud: gcloud compute instances list --filter="labels.osmedeus:*" ``` ## Cost Optimization 1. **Use preemptible instances** for all non-critical scans (`use_preemptible: true`) -- up to 80% savings 2. **Right-size machines**: e2-medium is enough for most single-target scans 3. **Always use `--auto-destroy`** to prevent forgotten instances 4. **Set cost limits** to catch runaway spending 5. **Use custom images** to reduce setup time (less instance-hours) 6. **Pick the cheapest region** if target geo-location doesn't matter (us-central1 is usually cheapest) 7. **GCP sustained-use discounts** apply automatically for on-demand VMs running more than 25% of the month # Hetzner Provider Guide Source: https://docs.osmedeus.org/cloud/provider-hetzner Step-by-step guide for running osmedeus cloud on Hetzner Cloud servers, the lowest cost provider web-ui-assets Introduction Cloud Step-by-step guide for running osmedeus cloud on Hetzner Cloud servers. Hetzner offers the lowest cost per instance among all supported providers, making it ideal for high-volume scanning. ## Prerequisites * A Hetzner Cloud account ([https://console.hetzner.cloud](https://console.hetzner.cloud)) * An API token * An SSH key pair (local `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`) ### Get Your API Token 1. Go to **Hetzner Cloud Console** > select your project (or create one) 2. **Security** > **API Tokens** > **Generate API Token** 3. Set permissions to **Read & Write** 4. Copy the token (shown only once) You can also store it as an environment variable: ```bash theme={null} export HETZNER_API_TOKEN="your-token-here" ``` ## Configuration ### Minimal Setup ```bash theme={null} # Enable cloud feature osmedeus config set cloud.enabled true # Credentials osmedeus cloud config set providers.hetzner.token ${HETZNER_API_TOKEN} osmedeus cloud config set providers.hetzner.location hel1 osmedeus cloud config set providers.hetzner.server_type "cx23" # 2 vCPU, 4GB RAM (current generation) osmedeus cloud config set defaults.provider hetzner # SSH osmedeus cloud config set ssh.private_key_path ~/.ssh/id_rsa osmedeus cloud config set ssh.public_key_path ~/.ssh/id_rsa.pub osmedeus cloud config set ssh.user root # Clean the setup scripts first osmedeus cloud config set setup.commands.clear "" # Worker setup osmedeus cloud config set setup.commands.add "sudo apt-get update" osmedeus cloud config set setup.commands.add "sudo apt-get install -y -qq curl git tmux unzip jq rsync" osmedeus cloud config set setup.commands.add "curl -fsSL https://www.osmedeus.org/install.sh | bash" osmedeus cloud config set setup.commands.add "osmedeus install base --preset" ``` ### Server Types Hetzner's pricing is significantly cheaper than other providers: | Server Type | vCPU | RAM | Disk | \$/hr | \$/month | Best For | | ----------- | ---- | ----- | ------ | --------- | --------- | --------------------------- | | cx22 | 2 | 4 GB | 40 GB | \~\$0.007 | \~\$4.50 | Light scans, single targets | | cx32 | 4 | 8 GB | 80 GB | \~\$0.013 | \~\$8.50 | General scanning | | cx42 | 8 | 16 GB | 160 GB | \~\$0.025 | \~\$16.50 | Heavy scans, parallel tools | | cx52 | 16 | 32 GB | 320 GB | \~\$0.050 | \~\$33.00 | Large-scale operations | | cpx21 | 3 | 4 GB | 80 GB | \~\$0.008 | \~\$5.50 | CPU-optimized scanning | | cpx31 | 4 | 8 GB | 160 GB | \~\$0.015 | \~\$10.00 | CPU-optimized, more RAM | ```bash theme={null} osmedeus cloud config set providers.hetzner.server_type cx32 ``` ### Locations | Location | Code | Region | | ----------- | ------ | ------- | | Falkenstein | `fsn1` | Germany | | Nuremberg | `nbg1` | Germany | | Helsinki | `hel1` | Finland | | Ashburn | `ash` | US East | | Hillsboro | `hil` | US West | | Singapore | `sin` | Asia | ```bash theme={null} osmedeus cloud config set providers.hetzner.location fsn1 ``` ### Custom Image Use a pre-built snapshot for faster boot: ```bash theme={null} # After setting up a server manually with all tools: # Hetzner Console > Servers > your-server > Snapshots > Create Snapshot # Note the snapshot ID osmedeus cloud config set providers.hetzner.image 12345678 ``` ### SSH Key (optional) If you have an SSH key registered in Hetzner Cloud: ```bash theme={null} # Hetzner Console > Security > SSH Keys > note the key name osmedeus cloud config set providers.hetzner.ssh_key_name my-key-name ``` ### Cost Limits ```bash theme={null} osmedeus cloud config set limits.max_hourly_spend 0.50 osmedeus cloud config set limits.max_total_spend 5.00 osmedeus cloud config set limits.max_instances 20 ``` ## Examples ### Quick Domain Recon ```bash theme={null} osmedeus cloud run -f fast -t example.com --auto-destroy ``` Cost: \~\$0.007 (1 x cx22 x 1 hour) -- less than a penny. ### Budget Bulk Scanning Hetzner's low prices make it perfect for scanning many targets: ```bash theme={null} # 20 workers scanning 200 targets for ~$0.28 osmedeus cloud run \ -f fast -T targets.txt --instances 20 \ --sync-back --auto-destroy ``` Cost: 20 x $0.007 x 2 hours = **$0.28\*\* ### Custom Command Pipeline ```bash theme={null} osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -o /tmp/osm-custom/live.txt" \ --custom-cmd "cat /tmp/osm-custom/live.txt | nuclei -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy ``` Cost: \~\$0.007 ### Distributed Nuclei at Scale ```bash theme={null} # Split 10,000 URLs across 10 Hetzner workers osmedeus cloud run \ --custom-cmd "cat {{Target}} | nuclei -o /tmp/osm-custom/results.txt" \ --sync-path "/tmp/osm-custom/results.txt" \ --sync-dest "./nuclei-hetzner" \ -T urls.txt --instances 10 --auto-destroy ``` Cost: 10 x $0.007 x 1 hour = **$0.07\*\* ### Port Scanning ```bash theme={null} # Use a bigger instance for masscan (needs more resources) osmedeus cloud config set providers.hetzner.server_type cx32 osmedeus cloud run \ --custom-cmd "masscan {{Target}} -p1-65535 --rate 10000 -oG /tmp/osm-custom/masscan.txt" \ --custom-cmd "cat /tmp/osm-custom/masscan.txt | grep 'Host:' | awk '{print \$2\":\" \$5}' | sed 's|/.*||' > /tmp/osm-custom/open-ports.txt" \ --sync-path "/tmp/osm-custom/" \ -t 203.0.113.0/24 --auto-destroy ``` ### Persistent Low-Cost Lab ```bash theme={null} # Create 5 workers and keep them running all day osmedeus cloud create --provider hetzner -n 5 # Run multiple scans throughout the day osmedeus cloud run -f fast -t target1.com --reuse osmedeus cloud run --custom-cmd "nmap -sV {{Target}}" -t target2.com --reuse osmedeus cloud run -f general -T targets.txt --reuse # Destroy at end of day osmedeus cloud destroy all --force ``` Cost: 5 x $0.007 x 8 hours = **$0.28\*\* for a full day of scanning on 5 machines. ### EU-Based Scanning Hetzner's European locations are useful when you need scans originating from EU IP space: ```bash theme={null} # Use German datacenter osmedeus cloud config set providers.hetzner.location fsn1 osmedeus cloud run -f fast -t eu-target.com --auto-destroy # Use Finnish datacenter osmedeus cloud config set providers.hetzner.location hel1 osmedeus cloud run -f fast -t nordic-target.com --auto-destroy ``` ### High-Performance with cx42 ```bash theme={null} # Use 8 vCPU / 16 GB RAM instance for heavy parallel scanning osmedeus cloud config set providers.hetzner.server_type cx42 osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -all -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -td -threads 200 -o /tmp/osm-custom/live.txt" \ --custom-cmd "cat /tmp/osm-custom/live.txt | nuclei -c 100 -o /tmp/osm-custom/nuclei.txt" \ --custom-cmd "cat /tmp/osm-custom/live.txt | katana -d 3 -jc -o /tmp/osm-custom/crawl.txt" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy ``` Cost: \~\$0.025 per hour ## Cost Comparison Why Hetzner is the cheapest option for bulk scanning: | Scenario | Hetzner (cx22) | DigitalOcean (s-2vcpu-4gb) | AWS (t3.medium) | | ---------------------- | -------------- | -------------------------- | --------------- | | 1 instance x 1 hour | \$0.007 | \$0.022 | \$0.042 | | 5 instances x 2 hours | \$0.07 | \$0.22 | \$0.42 | | 10 instances x 4 hours | \$0.28 | \$0.89 | \$1.66 | | 20 instances x 8 hours | \$1.12 | \$3.57 | \$6.66 | Hetzner is \~3x cheaper than DigitalOcean and \~6x cheaper than AWS for equivalent specs. ## Troubleshooting ### "Unauthorized" Error Your API token is invalid or expired. Generate a new one in the Hetzner Cloud Console. ```bash theme={null} osmedeus cloud config set providers.hetzner.token ``` ### Server Type Not Available Some server types may not be available in all locations. Try a different location: ```bash theme={null} osmedeus cloud config set providers.hetzner.location nbg1 ``` ### SSH Connection Issues Hetzner servers default to `root` user: ```bash theme={null} osmedeus cloud config set ssh.user root ``` Verify your SSH key is correctly configured: ```bash theme={null} osmedeus cloud run --custom-cmd "whoami" -t test --verbose-setup ``` ### Rate Limiting Hetzner's API has rate limits. If creating many instances at once, you may hit them. Space out creation or contact Hetzner support to increase limits. ### Cleaning Up ```bash theme={null} # List all infrastructure osmedeus cloud list # Destroy specific osmedeus cloud destroy # Destroy everything osmedeus cloud destroy all --force # If out of sync, check Hetzner Console directly: # Console > Servers > look for osmedeus-prefixed servers ``` ## Best Practices 1. **Use cx22 as default** -- 2 vCPU / 4 GB is enough for most scans at \$0.007/hr 2. **Scale horizontally** -- 10 x cx22 is cheaper and faster than 1 x cx52 for parallelizable workloads 3. **Always `--auto-destroy`** -- even at \$0.007/hr, forgotten instances add up 4. **Use European locations** (fsn1, nbg1) for lowest latency to Hetzner's network 5. **Pre-build snapshots** for frequently-used tool configurations to skip setup time 6. **Set modest cost limits** -- even \$5.00 max\_total\_spend goes a long way at Hetzner pricing # Cloud Quick Reference Source: https://docs.osmedeus.org/cloud/quick-reference Quick reference for cloud configuration, infrastructure management, and command flags ## Setup (30 seconds) ```bash theme={null} # Enable cloud feature osmedeus config set cloud.enabled true # Set credentials (pick your provider) osmedeus cloud config set providers.aws.access_key_id ${AWS_ACCESS_KEY_ID} osmedeus cloud config set providers.aws.secret_access_key ${AWS_SECRET_ACCESS_KEY} osmedeus cloud config set providers.aws.region ap-southeast-1 osmedeus cloud config set defaults.provider aws # SSH keys osmedeus cloud config set ssh.private_key_path ~/.ssh/id_rsa osmedeus cloud config set ssh.public_key_path ~/.ssh/id_rsa.pub # Clean the setup scripts first osmedeus cloud config set setup.commands.clear "" # Worker setup commands osmedeus cloud config set setup.commands.add "curl -fsSL https://www.osmedeus.org/install.sh | bash" osmedeus cloud config set setup.commands.add "osmedeus install base --preset" ``` ## Configuration ```bash theme={null} osmedeus cloud config list # View all settings osmedeus cloud config set # Set a value osmedeus cloud config set .add # Append to list osmedeus cloud config clean # Reset to defaults # Provider credentials osmedeus cloud config set providers.. # Instance type osmedeus cloud config set providers.aws.instance_type t3.large # Spot instances (70-80% cheaper) osmedeus cloud config set providers.aws.use_spot true # Cost limits osmedeus cloud config set limits.max_hourly_spend 1.00 osmedeus cloud config set limits.max_total_spend 10.00 osmedeus cloud config set limits.max_instances 10 ``` ## Infrastructure ```bash theme={null} osmedeus cloud create --provider aws -n 3 # Create instances osmedeus cloud list # List active infra osmedeus cloud destroy # Destroy by ID osmedeus cloud destroy all --force # Destroy everything osmedeus cloud setup --reuse-with "1.2.3.4,5.6.7.8" # Setup existing machines ``` ## Workflow Mode ```bash theme={null} # Basic osmedeus cloud run -f fast -t example.com osmedeus cloud run -m enum-subdomain -t example.com --timeout 30m # Multiple instances osmedeus cloud run -f general -t example.com --instances 3 --provider aws # Multiple targets distributed across workers osmedeus cloud run -f fast -T targets.txt --instances 5 osmedeus cloud run -f fast -T targets.txt --chunk-size 10 # 10 targets per worker osmedeus cloud run -f fast -T targets.txt --chunk-count 3 # Split into 3 chunks # Reuse existing infrastructure osmedeus cloud run -f fast -t example.com --reuse osmedeus cloud run -f fast -t example.com --reuse-with "1.2.3.4,5.6.7.8" # Sync results back + auto-destroy osmedeus cloud run -f fast -t example.com --sync-back --auto-destroy ``` ## Custom Command Mode Run arbitrary commands on cloud instances (mutually exclusive with `-f`/`-m`): ```bash theme={null} # Single command osmedeus cloud run --custom-cmd "nmap -sV {{Target}}" -t example.com # Multiple sequential commands osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -o /tmp/osm-custom/live.txt" \ -t example.com # Post-commands (run only if all custom-cmds succeed) osmedeus cloud run \ --custom-cmd "nuclei -u {{Target}} -o /tmp/osm-custom/results.txt" \ --custom-post-cmd "cat /tmp/osm-custom/results.txt | notify" \ -t example.com # Sync results back osmedeus cloud run \ --custom-cmd "nmap -sV {{Target}} -oA /tmp/osm-custom/scan" \ --sync-path "/tmp/osm-custom/" \ --sync-dest "./my-results" \ -t example.com # Distribute targets across workers osmedeus cloud run \ --custom-cmd "cat {{Target}} | httpx -o /tmp/osm-custom/live.txt" \ --sync-path "/tmp/osm-custom/live.txt" \ -T targets.txt --instances 5 --auto-destroy ``` ### Template Variables | Variable | Description | | ----------------- | -------------------------------------------- | | `{{Target}}` | Target string or chunk file path (with `-T`) | | `{{public_ip}}` | Worker's public IP | | `{{private_ip}}` | Worker's private IP | | `{{worker_name}}` | Resource name | | `{{worker_id}}` | Cloud resource ID | | `{{infra_id}}` | Infrastructure ID | | `{{provider}}` | Provider name | | `{{ssh_user}}` | SSH username | | `{{index}}` | Worker index (0, 1, 2, ...) | ### Behavior * Commands run in `/tmp/osm-custom/` on the remote * Custom-cmds run sequentially per worker, in parallel across workers * First failure stops remaining commands and skips post-cmds for that worker * Sync downloads to: `/-/` ## Flags Reference | Flag | Short | Description | | ------------------- | ----- | ------------------------------------------------- | | `--flow` | `-f` | Flow workflow name | | `--module` | `-m` | Module workflow name | | `--target` | `-t` | Single target | | `--target-file` | `-T` | File containing targets | | `--provider` | `-p` | Cloud provider | | `--instances` | `-n` | Number of instances | | `--timeout` | | Scan timeout (e.g., `2h`, `30m`) | | `--auto-destroy` | | Destroy infrastructure after completion | | `--reuse` | | Auto-discover existing infrastructure | | `--reuse-with` | | Reuse specific IPs (comma-separated) | | `--sync-back` | | Download workflow results (workflow mode) | | `--verbose-setup` | | Show full setup command output | | `--ansible` | | Use Ansible playbook for setup | | `--chunk-size` | | Targets per worker chunk | | `--chunk-count` | | Split targets into N chunks | | `--custom-cmd` | | Custom command (repeatable) | | `--custom-post-cmd` | | Post-command (repeatable) | | `--sync-path` | | Remote path to download (repeatable) | | `--sync-dest` | | Local sync directory (default: `./osm-sync-back`) | ## Cost Reference | Provider | Instance | vCPU | RAM | Hourly | | ------------ | ------------- | ---- | ------ | --------- | | Hetzner | cx22 | 2 | 4 GB | \~\$0.007 | | Linode | g6-standard-2 | 2 | 4 GB | \$0.018 | | DigitalOcean | s-2vcpu-4gb | 2 | 4 GB | \$0.02232 | | AWS | t3.medium | 2 | 4 GB | \$0.0416 | | GCP | n1-standard-2 | 2 | 7.5 GB | \$0.095 | | Azure | Standard\_B2s | 2 | 4 GB | \$0.042 | # Cloud Usage Examples Source: https://docs.osmedeus.org/cloud/usage-examples Practical examples for running distributed security scans and custom commands on cloud infrastructure Practical examples for running distributed security scans and custom commands on cloud infrastructure. ## Getting Started ### First Scan in 5 Minutes ```bash theme={null} # Step 0: Enable cloud feature osmedeus config set cloud.enabled true # Step 1: Configure AWS credentials osmedeus cloud config set providers.aws.access_key_id ${AWS_ACCESS_KEY_ID} osmedeus cloud config set providers.aws.secret_access_key ${AWS_SECRET_ACCESS_KEY} osmedeus cloud config set providers.aws.region ap-southeast-1 osmedeus cloud config set defaults.provider aws # Step 2: SSH keys osmedeus cloud config set ssh.private_key_path ~/.ssh/id_rsa osmedeus cloud config set ssh.public_key_path ~/.ssh/id_rsa.pub # Step 3: Clean the setup scripts first, then add setup commands for workers osmedeus cloud config set setup.commands.clear "" osmedeus cloud config set setup.commands.add "curl -fsSL https://www.osmedeus.org/install.sh | bash" osmedeus cloud config set setup.commands.add "osmedeus install base --preset" # Step 4: Run your first scan osmedeus cloud run -f fast -t example.com --auto-destroy ``` This provisions 1 AWS instance, installs osmedeus + tools, runs the `fast` flow against `example.com`, streams output to your terminal, and destroys the instance when done. ### Verify Configuration ```bash theme={null} # View all settings osmedeus cloud config list # View with secrets visible osmedeus cloud config list --show-secrets ``` ## Workflow Mode Examples ### Single Target ```bash theme={null} # Run a flow osmedeus cloud run -f fast -t example.com # Run a specific module osmedeus cloud run -m enum-subdomain -t example.com # With timeout osmedeus cloud run -f general -t example.com --timeout 2h # With specific provider osmedeus cloud run -f fast -t example.com --provider digitalocean ``` ### Multiple Targets ```bash theme={null} # Distribute targets across 5 workers osmedeus cloud run -f fast -T targets.txt --instances 5 # 10 targets per worker (auto-calculates worker count) osmedeus cloud run -f fast -T targets.txt --instances 10 --chunk-size 10 # Split into exactly 3 chunks osmedeus cloud run -f fast -T targets.txt --instances 5 --chunk-count 3 ``` ### Full Lifecycle ```bash theme={null} # Provision, scan, sync results back, then destroy osmedeus cloud run -f fast -t example.com --sync-back --auto-destroy # Same with multiple targets osmedeus cloud run -f fast -T targets.txt --instances 3 --sync-back --auto-destroy ``` ### Reusing Infrastructure ```bash theme={null} # First run: provision and scan osmedeus cloud run -f fast -t target1.com # Second run: reuse same instances for a different target osmedeus cloud run -f fast -t target2.com --reuse # Reuse specific machines by IP osmedeus cloud run -f fast -t target3.com --reuse-with "1.2.3.4,5.6.7.8" # When done, destroy manually osmedeus cloud destroy ``` ## Custom Command Examples ### Basic Usage ```bash theme={null} # Run a single command osmedeus cloud run --custom-cmd "nmap -sV {{Target}}" -t example.com # Run on existing infrastructure osmedeus cloud run --custom-cmd "whoami && id" -t example.com --reuse ``` ### Recon Pipeline ```bash theme={null} # Subdomain enumeration → HTTP probing → screenshot osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -o /tmp/osm-custom/live.txt" \ --custom-cmd "cat /tmp/osm-custom/live.txt | gowitness scan -o /tmp/osm-custom/screenshots" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy ``` ### Vulnerability Scanning ```bash theme={null} # Nuclei scan with custom templates osmedeus cloud run \ --custom-cmd "nuclei -u {{Target}} -t cves/ -o /tmp/osm-custom/cves.txt" \ --custom-cmd "nuclei -u {{Target}} -t exposures/ -o /tmp/osm-custom/exposures.txt" \ --custom-post-cmd "cat /tmp/osm-custom/cves.txt /tmp/osm-custom/exposures.txt | sort -u > /tmp/osm-custom/all-findings.txt" \ --sync-path "/tmp/osm-custom/all-findings.txt" \ -t example.com ``` ### Port Scanning at Scale ```bash theme={null} # Distribute an IP list across 10 workers for masscan + nmap osmedeus cloud run \ --custom-cmd "cat {{Target}} | while read ip; do masscan \$ip -p1-65535 --rate 1000 -oG /tmp/osm-custom/masscan-\$(echo \$ip | tr '.' '-').txt; done" \ --custom-post-cmd "cat /tmp/osm-custom/masscan-*.txt > /tmp/osm-custom/all-ports.txt" \ --sync-path "/tmp/osm-custom/" \ -T ip-list.txt --instances 10 --auto-destroy ``` ### SAST Scanning ```bash theme={null} # Clone a repo and run semgrep osmedeus cloud run \ --custom-cmd "git clone https://github.com/org/repo.git /tmp/osm-custom/repo" \ --custom-cmd "semgrep --config auto /tmp/osm-custom/repo --sarif -o /tmp/osm-custom/semgrep.sarif" \ --sync-path "/tmp/osm-custom/semgrep.sarif" \ -t org/repo --auto-destroy ``` ### Custom Sync Destination ```bash theme={null} # Download to a specific local directory osmedeus cloud run \ --custom-cmd "nmap -sV {{Target}} -oA /tmp/osm-custom/nmap" \ --sync-path "/tmp/osm-custom/" \ --sync-dest "./nmap-results" \ -t example.com # Results land in: ./nmap-results/-/tmp/osm-custom/nmap.* ``` ### Using Worker Variables ```bash theme={null} # Log worker info alongside scan results osmedeus cloud run \ --custom-cmd "echo 'Worker {{worker_name}} ({{public_ip}}) scanning {{Target}}' > /tmp/osm-custom/info.txt" \ --custom-cmd "nmap -sV {{Target}} -oA /tmp/osm-custom/nmap" \ --sync-path "/tmp/osm-custom/" \ -t example.com ``` ## Real-World Scenarios ### Bug Bounty: Enumerate Multiple Programs ```bash theme={null} # targets.txt contains: hackerone.com, bugcrowd.com, intigriti.com, ... osmedeus cloud run \ -f general -T targets.txt --instances 5 \ --sync-back --auto-destroy --provider digitalocean ``` ### Scan a Large IP Range ```bash theme={null} # ip-ranges.txt contains CIDR ranges, one per line osmedeus cloud run \ --custom-cmd "cat {{Target}} | nmap -iL - -sV -oA /tmp/osm-custom/scan" \ --sync-path "/tmp/osm-custom/" \ -T ip-ranges.txt --instances 10 --auto-destroy ``` ### Persistent Campaign ```bash theme={null} # Create infrastructure once osmedeus cloud create --provider aws -n 3 # Run multiple scans over time osmedeus cloud run -f fast -t target1.com --reuse osmedeus cloud run -f fast -t target2.com --reuse osmedeus cloud run --custom-cmd "nuclei -u target3.com -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/" -t target3.com --reuse # Destroy when the campaign is over osmedeus cloud destroy all --force ``` ### Multi-Provider Strategy ```bash theme={null} # Use Hetzner for cheap bulk scanning osmedeus cloud run -f fast -T targets.txt --instances 10 --provider hetzner # Use AWS for targets requiring specific geo-location osmedeus cloud run -f fast -t us-target.com --provider aws ``` ## Provider Configuration Examples ### AWS ```bash theme={null} osmedeus cloud config set providers.aws.access_key_id ${AWS_ACCESS_KEY_ID} osmedeus cloud config set providers.aws.secret_access_key ${AWS_SECRET_ACCESS_KEY} osmedeus cloud config set providers.aws.region ap-southeast-1 osmedeus cloud config set providers.aws.instance_type t3.medium osmedeus cloud config set providers.aws.use_spot true osmedeus cloud config set defaults.provider aws ``` See [AWS Provider Guide](/cloud/provider-aws) for detailed setup and examples. ### Hetzner ```bash theme={null} osmedeus cloud config set providers.hetzner.token ${HETZNER_API_TOKEN} osmedeus cloud config set providers.hetzner.location fsn1 osmedeus cloud config set providers.hetzner.server_type cx22 osmedeus cloud config set defaults.provider hetzner ``` See [Hetzner Provider Guide](/cloud/provider-hetzner) for detailed setup and examples. ### DigitalOcean ```bash theme={null} osmedeus cloud config set providers.digitalocean.token ${DO_TOKEN} osmedeus cloud config set providers.digitalocean.region sgp1 osmedeus cloud config set providers.digitalocean.size s-2vcpu-4gb osmedeus cloud config set defaults.provider digitalocean ``` ### GCP ```bash theme={null} osmedeus cloud config set providers.gcp.project_id ${GCP_PROJECT} osmedeus cloud config set providers.gcp.credentials_file /path/to/sa-key.json osmedeus cloud config set providers.gcp.region us-central1 osmedeus cloud config set providers.gcp.zone us-central1-a osmedeus cloud config set providers.gcp.machine_type n1-standard-2 osmedeus cloud config set providers.gcp.use_preemptible true osmedeus cloud config set defaults.provider gcp ``` ### Linode ```bash theme={null} osmedeus cloud config set providers.linode.token ${LINODE_TOKEN} osmedeus cloud config set providers.linode.region ap-south osmedeus cloud config set providers.linode.type g6-standard-2 osmedeus cloud config set defaults.provider linode ``` ### Azure ```bash theme={null} osmedeus cloud config set providers.azure.subscription_id ${AZURE_SUB_ID} osmedeus cloud config set providers.azure.tenant_id ${AZURE_TENANT_ID} osmedeus cloud config set providers.azure.client_id ${AZURE_CLIENT_ID} osmedeus cloud config set providers.azure.client_secret ${AZURE_CLIENT_SECRET} osmedeus cloud config set providers.azure.location southeastasia osmedeus cloud config set providers.azure.vm_size Standard_B2s osmedeus cloud config set defaults.provider azure ``` ## Advanced Topics ### Custom Snapshots Pre-install tools on a VM, snapshot it, then use the snapshot for faster boot: ```bash theme={null} # 1. Create and set up a VM manually via your provider's console # 2. Install osmedeus + all tools # 3. Create a snapshot/image in the provider console # 4. Configure osmedeus to use it # AWS osmedeus cloud config set providers.aws.ami ami-0123456789abcdef0 # DigitalOcean osmedeus cloud config set providers.digitalocean.snapshot_id 12345678 # Hetzner osmedeus cloud config set providers.hetzner.image 12345678 ``` Boot time drops from \~5 minutes to \~30 seconds. ### Custom Worker Setup ```bash theme={null} # Add setup commands (run in order on each worker) osmedeus cloud config set setup.commands.add "apt-get update && apt-get install -y nmap masscan" osmedeus cloud config set setup.commands.add "go install github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest" # Add post-setup commands (with per-worker variable expansion) osmedeus cloud config set setup.post_commands.add "echo '{{worker_name}} at {{public_ip}}' >> /tmp/workers.txt" ``` ### Ansible Setup ```bash theme={null} osmedeus cloud config set setup.ansible.enabled true osmedeus cloud config set setup.ansible.playbook_path /path/to/setup.yaml osmedeus cloud run -f fast -t example.com --ansible ``` ### Environment Variable Expansion All config values support `${ENV_VAR}` syntax: ```bash theme={null} osmedeus cloud config set providers.aws.access_key_id '${AWS_ACCESS_KEY_ID}' osmedeus cloud config set providers.aws.secret_access_key '${AWS_SECRET_ACCESS_KEY}' ``` Values are expanded at runtime from your shell environment. ## Troubleshooting ### Workers not connecting ```bash theme={null} osmedeus cloud run -f fast -t example.com --verbose-setup # See SSH output osmedeus cloud run -f fast -t example.com --debug # Full debug logs ``` ### Orphaned infrastructure ```bash theme={null} osmedeus cloud list # Check what's running osmedeus cloud destroy all --force # Emergency cleanup ``` ### Cost limit exceeded ```bash theme={null} osmedeus cloud config set limits.max_hourly_spend 5.00 # Increase limit ``` ### Custom command failed * Check if the tool is installed in your setup commands * Use `--verbose-setup` to verify setup completed * Test with a simple command first: `--custom-cmd "which nmap"` # Cloud Usage Guide Source: https://docs.osmedeus.org/cloud/usage-guide Provision cloud infrastructure and run security workflows or arbitrary commands on cloud VMs Introduction Cloud web-ui-vuln Osmedeus Cloud provisions virtual machines across cloud providers and runs security workflows or arbitrary commands on them. This guide covers the architecture, configuration, and operational patterns. ## How It Works ``` Local Machine Cloud Provider ┌──────────────┐ ┌──────────────────┐ │ osmedeus │ 1. Provision │ Worker VM 1 │ │ cloud run │ ──────────────────► │ ┌────────────┐ │ │ │ 2. SSH setup │ │ osmedeus │ │ │ │ ──────────────────► │ │ + tools │ │ │ │ 3. Stream output │ └────────────┘ │ │ │ ◄────────────────── │ │ │ │ ├──────────────────┤ │ │ (same for each) │ Worker VM 2 │ │ │ ◄──────────────────► │ ... │ │ │ ├──────────────────┤ │ │ 4. Sync results │ Worker VM N │ │ │ ◄────────────────── │ ... │ │ │ 5. Destroy └──────────────────┘ └──────────────┘ ``` **Lifecycle:** 1. **Provision** -- Create VMs via Pulumi (or reuse existing ones) 2. **Setup** -- SSH into each worker, run setup commands (install osmedeus, tools, etc.) 3. **Execute** -- Run workflow or custom commands, stream output back in real time 4. **Sync** -- Download results to local machine (optional) 5. **Destroy** -- Tear down infrastructure (optional, can be automatic) ## Supported Providers | Provider | Config Key | Instance Types | | ------------ | -------------- | -------------------------------------- | | AWS | `aws` | t3.medium, t3.large, t3.xlarge | | DigitalOcean | `digitalocean` | s-2vcpu-4gb, s-4vcpu-8gb, s-8vcpu-16gb | | GCP | `gcp` | n1-standard-2, n1-standard-4 | | Hetzner | `hetzner` | cx22, cx32, cx42 | | Linode | `linode` | g6-standard-2, g6-standard-4 | | Azure | `azure` | Standard\_B2s, Standard\_D2s\_v3 | ## Configuration Cloud config lives in `~/.osmedeus/cloud/cloud-settings.yaml`. Manage it with: ```bash theme={null} # Set a value osmedeus cloud config set # View current config osmedeus cloud config list # Reset to defaults osmedeus cloud config clean ``` ### Required Configuration Every provider needs four things: **cloud enabled**, **credentials**, **SSH keys**, and **setup commands**. ```bash theme={null} # 0. Enable cloud feature osmedeus config set cloud.enabled true # 1. Provider credentials (example: AWS) osmedeus cloud config set providers.aws.access_key_id ${AWS_ACCESS_KEY_ID} osmedeus cloud config set providers.aws.secret_access_key ${AWS_SECRET_ACCESS_KEY} osmedeus cloud config set providers.aws.region ap-southeast-1 # 2. SSH keys (used to connect to workers) osmedeus cloud config set ssh.private_key_path ~/.ssh/id_rsa osmedeus cloud config set ssh.public_key_path ~/.ssh/id_rsa.pub # 3. Clean the setup scripts first, then add setup commands (run on each worker before scanning) osmedeus cloud config set setup.commands.clear "" osmedeus cloud config set setup.commands.add "curl -fsSL https://www.osmedeus.org/install.sh | bash" osmedeus cloud config set setup.commands.add "osmedeus install base --preset" # 4. Set default provider osmedeus cloud config set defaults.provider aws ``` ### Optional Configuration ```bash theme={null} # Instance type osmedeus cloud config set providers.aws.instance_type t3.large # Use spot/preemptible instances (70-80% cheaper) osmedeus cloud config set providers.aws.use_spot true # Cost limits osmedeus cloud config set limits.max_hourly_spend 1.00 osmedeus cloud config set limits.max_total_spend 10.00 osmedeus cloud config set limits.max_instances 10 # Default timeout osmedeus cloud config set defaults.timeout 2h # SSH user (default: root for most providers, ubuntu for AWS) osmedeus cloud config set ssh.user root ``` ### Post-Setup Commands Post-setup commands run per-worker after the main setup, with template variables expanded: ```bash theme={null} osmedeus cloud config set setup.post_commands.add "echo 'Worker {{index}} ready at {{public_ip}}'" ``` Available variables: `{{public_ip}}`, `{{private_ip}}`, `{{worker_name}}`, `{{worker_id}}`, `{{infra_id}}`, `{{provider}}`, `{{ssh_user}}`, `{{index}}` ## Two Execution Modes ### Workflow Mode (default) Runs an osmedeus flow or module on remote workers: ```bash theme={null} # Run a flow osmedeus cloud run -f fast -t example.com # Run a module osmedeus cloud run -m enum-subdomain -t example.com ``` ### Custom Command Mode Runs arbitrary shell commands on remote workers -- no osmedeus workflow required: ```bash theme={null} osmedeus cloud run --custom-cmd "nmap -sV {{Target}} -oA /tmp/osm-custom/nmap" -t example.com ``` `--custom-cmd` is mutually exclusive with `-f`/`-m`. See [Custom Command Mode](#custom-command-mode-details) below. ## Infrastructure Management ### Provisioning ```bash theme={null} # Provision with cloud run (creates + runs + optional destroy) osmedeus cloud run -f fast -t example.com --instances 3 # Provision separately (no scan) osmedeus cloud create --provider aws -n 3 ``` ### Listing ```bash theme={null} osmedeus cloud list ``` ### Reusing Existing Infrastructure ```bash theme={null} # Auto-discover from saved state osmedeus cloud run -f fast -t example.com --reuse # Specify IPs directly osmedeus cloud run -f fast -t example.com --reuse-with "1.2.3.4,5.6.7.8" ``` ### Destroying ```bash theme={null} # Destroy specific infrastructure osmedeus cloud destroy # Destroy all osmedeus cloud destroy all --force ``` ## Target Distribution When scanning multiple targets across multiple workers, osmedeus splits the target list into chunks: ```bash theme={null} # 100 targets across 5 workers = 20 targets each osmedeus cloud run -f fast -T targets.txt --instances 5 # Control chunk size: 10 targets per worker osmedeus cloud run -f fast -T targets.txt --instances 10 --chunk-size 10 # Control chunk count: split into exactly 3 chunks osmedeus cloud run -f fast -T targets.txt --instances 5 --chunk-count 3 ``` Each worker receives its chunk as a file at `/tmp/osm-targets-{i}.txt` on the remote machine. ## Custom Command Mode Details Run any commands on cloud instances without using osmedeus workflows. Commands run in `/tmp/osm-custom/` on the remote. ### Flags | Flag | Description | | ------------------- | --------------------------------------------------------------- | | `--custom-cmd` | Command to run (repeatable, sequential per worker) | | `--custom-post-cmd` | Runs after all custom-cmds succeed (repeatable) | | `--sync-path` | Remote path to download after execution (repeatable) | | `--sync-dest` | Local base directory for downloads (default: `./osm-sync-back`) | ### Template Variables All commands and sync paths support these variables: | Variable | Description | Example | | ----------------- | ------------------------------------------- | ----------------------------------------- | | `{{Target}}` | Target string, or chunk file path with `-T` | `example.com` or `/tmp/osm-targets-0.txt` | | `{{public_ip}}` | Worker's public IP | `203.0.113.10` | | `{{private_ip}}` | Worker's private IP | `10.0.0.5` | | `{{worker_name}}` | Resource name | `osmw-1775159841-0` | | `{{worker_id}}` | Cloud resource ID | `i-0437adf5...` | | `{{infra_id}}` | Infrastructure ID | `cloud-aws-1775159841` | | `{{provider}}` | Provider name | `aws` | | `{{ssh_user}}` | SSH username | `ubuntu` | | `{{index}}` | Worker index | `0`, `1`, `2` | ### Execution Rules * Custom-cmds run **sequentially** on each worker, but **in parallel** across workers * If any `--custom-cmd` fails (non-zero exit), remaining commands and all `--custom-post-cmd` are skipped for that worker * Post-cmd failures are logged but do not affect other workers ### Sync-Back Downloaded files are placed at: `/-/` For example, `--sync-path /tmp/osm-custom/results.txt` from worker `osmw-0` at `1.2.3.4`: ``` ./osm-sync-back/osmw-0-1.2.3.4/tmp/osm-custom/results.txt ``` ### Examples ```bash theme={null} # Simple: run nmap on a cloud instance osmedeus cloud run \ --custom-cmd "nmap -sV {{Target}} -oA /tmp/osm-custom/nmap-result" \ --sync-path "/tmp/osm-custom/" \ -t example.com --auto-destroy # Multi-step pipeline with post-processing osmedeus cloud run \ --custom-cmd "subfinder -d {{Target}} -o /tmp/osm-custom/subs.txt" \ --custom-cmd "cat /tmp/osm-custom/subs.txt | httpx -o /tmp/osm-custom/live.txt" \ --custom-post-cmd "wc -l /tmp/osm-custom/live.txt" \ --sync-path "/tmp/osm-custom/subs.txt" \ --sync-path "/tmp/osm-custom/live.txt" \ -t example.com # Distribute target list across 5 workers osmedeus cloud run \ --custom-cmd "cat {{Target}} | nuclei -o /tmp/osm-custom/nuclei.txt" \ --sync-path "/tmp/osm-custom/nuclei.txt" \ --sync-dest "./nuclei-results" \ -T targets.txt --instances 5 --auto-destroy ``` ## Syncing Results ### Workflow Mode: `--sync-back` Exports osmedeus workspaces (including database state) from remote workers and imports them locally: ```bash theme={null} osmedeus cloud run -f fast -t example.com --sync-back ``` ### Custom Mode: `--sync-path` Downloads specific files or directories via SFTP: ```bash theme={null} osmedeus cloud run --custom-cmd "..." --sync-path "/tmp/osm-custom/" -t example.com ``` ## Cost Management ### Pre-Provisioning Estimates Costs are estimated before provisioning. Set limits to prevent overspending: ```bash theme={null} osmedeus cloud config set limits.max_hourly_spend 1.00 osmedeus cloud config set limits.max_total_spend 10.00 osmedeus cloud config set limits.max_instances 10 ``` ### Spot/Preemptible Instances Save 70-80% on instance costs: ```bash theme={null} # AWS spot instances osmedeus cloud config set providers.aws.use_spot true # GCP preemptible instances osmedeus cloud config set providers.gcp.use_preemptible true ``` ### Cost Reference | Provider | Instance | vCPU | RAM | Hourly | | ------------ | ------------- | ---- | ------ | --------- | | Hetzner | cx22 | 2 | 4 GB | \~\$0.007 | | Linode | g6-standard-2 | 2 | 4 GB | \$0.018 | | DigitalOcean | s-2vcpu-4gb | 2 | 4 GB | \$0.02232 | | AWS | t3.medium | 2 | 4 GB | \$0.0416 | | GCP | n1-standard-2 | 2 | 7.5 GB | \$0.095 | | Azure | Standard\_B2s | 2 | 4 GB | \$0.042 | **Example:** 5 DigitalOcean s-2vcpu-4gb instances for 2 hours = 5 x $0.02232 x 2 = **$0.22\*\* ## Worker Setup Workers are set up via SSH after provisioning. The setup flow: 1. **Cloud-init** (automatic): Installs SSH keys, basic packages 2. **Setup commands** (`setup.commands`): Install osmedeus, tools, base data 3. **Post-setup commands** (`setup.post_commands`): Per-worker configuration with template variables ### Ansible Alternative For complex setups, use Ansible instead of SSH commands: ```bash theme={null} osmedeus cloud config set setup.ansible.enabled true osmedeus cloud config set setup.ansible.playbook_path /path/to/playbook.yaml osmedeus cloud run -f fast -t example.com --ansible ``` ### Setup on Existing Machines ```bash theme={null} osmedeus cloud setup --reuse-with "1.2.3.4,5.6.7.8" ``` ## Troubleshooting ### Workers Not Connecting ```bash theme={null} # Verbose setup to see SSH output osmedeus cloud run -f fast -t example.com --verbose-setup # Debug mode for full logging osmedeus cloud run -f fast -t example.com --debug ``` ### Infrastructure Stuck ```bash theme={null} # List all infrastructure osmedeus cloud list # Force destroy everything osmedeus cloud destroy all --force ``` ### Cost Exceeded If cost limits are hit, provisioning is blocked. Adjust limits: ```bash theme={null} osmedeus cloud config set limits.max_hourly_spend 5.00 ``` ## Best Practices 1. **Always set cost limits** before running large-scale scans 2. **Use `--auto-destroy`** to avoid forgotten instances accruing charges 3. **Use spot instances** for non-critical scans (70-80% savings) 4. **Use `--reuse`** to avoid re-provisioning for iterative work 5. **Start small** -- test with 1 instance before scaling up 6. **Use custom snapshots** with tools pre-installed to cut setup time from 5min to 30s 7. **Check `cloud list`** regularly to verify no orphaned infrastructure # Adding CLI Commands Source: https://docs.osmedeus.org/extending/cli-commands Add custom commands to the Osmedeus CLI. ## Overview CLI commands use Cobra and are defined in `pkg/cli/`. Each command file typically contains one main command with optional subcommands. ## Command Structure ```go theme={null} // pkg/cli/mycommand.go package cli import ( "fmt" "github.com/spf13/cobra" ) var ( // Command flags myFlag string myBoolFlag bool ) var myCmd = &cobra.Command{ Use: "mycommand", Short: "Short description", Long: `Long description with details.`, RunE: func(cmd *cobra.Command, args []string) error { return runMyCommand(args) }, } func init() { // Register with root command rootCmd.AddCommand(myCmd) // Add flags myCmd.Flags().StringVarP(&myFlag, "flag", "f", "", "Flag description") myCmd.Flags().BoolVarP(&myBoolFlag, "verbose", "v", false, "Verbose output") // Add subcommands myCmd.AddCommand(mySubCmd) } func runMyCommand(args []string) error { // Load configuration cfg, err := loadConfig() if err != nil { return err } // Command logic fmt.Printf("Running mycommand with flag: %s\n", myFlag) return nil } ``` ## Steps to Add a Command ### 1. Create Command File Create `pkg/cli/mycommand.go`: ```go theme={null} package cli import ( "fmt" "github.com/osmedeus/osmedeus-ng/internal/config" "github.com/osmedeus/osmedeus-ng/internal/terminal" "github.com/spf13/cobra" ) var ( targetFlag string outputFlag string verboseFlag bool ) var myCmd = &cobra.Command{ Use: "mycommand [subcommand]", Aliases: []string{"my", "mc"}, Short: "My custom command", Long: `My custom command does something useful. Examples: osmedeus mycommand do-something -t example.com osmedeus mycommand list`, RunE: func(cmd *cobra.Command, args []string) error { // Show help if no subcommand return cmd.Help() }, } var myDoCmd = &cobra.Command{ Use: "do-something", Short: "Do something specific", RunE: func(cmd *cobra.Command, args []string) error { return runDoSomething() }, } var myListCmd = &cobra.Command{ Use: "list", Aliases: []string{"ls"}, Short: "List items", RunE: func(cmd *cobra.Command, args []string) error { return runList() }, } func init() { // Register main command rootCmd.AddCommand(myCmd) // Add flags to main command (inherited by subcommands) myCmd.PersistentFlags().StringVarP(&targetFlag, "target", "t", "", "Target") myCmd.PersistentFlags().BoolVarP(&verboseFlag, "verbose", "v", false, "Verbose") // Add subcommand-specific flags myDoCmd.Flags().StringVarP(&outputFlag, "output", "o", "", "Output path") // Register subcommands myCmd.AddCommand(myDoCmd) myCmd.AddCommand(myListCmd) } func runDoSomething() error { cfg, err := config.Load(settingsFile) if err != nil { return fmt.Errorf("failed to load config: %w", err) } if targetFlag == "" { return fmt.Errorf("target is required") } terminal.PrintInfo("Processing target: %s", targetFlag) // Your logic here... terminal.PrintSuccess("Done!") return nil } func runList() error { cfg, err := config.Load(settingsFile) if err != nil { return err } items := []string{"item1", "item2", "item3"} terminal.PrintTable([]string{"Name", "Status"}, [][]string{ {"item1", "active"}, {"item2", "pending"}, {"item3", "complete"}, }) return nil } ``` ### 2. Use Terminal Helpers The `internal/terminal` package provides output formatting: ```go theme={null} import "github.com/osmedeus/osmedeus-ng/internal/terminal" // Print messages terminal.PrintInfo("Information message") terminal.PrintSuccess("Success message") terminal.PrintWarning("Warning message") terminal.PrintError("Error message") // Print formatted terminal.PrintInfo("Processing %s with %d threads", target, threads) // Print tables terminal.PrintTable( []string{"Column1", "Column2"}, [][]string{ {"row1-col1", "row1-col2"}, {"row2-col1", "row2-col2"}, }, ) // Print with colors terminal.PrintColored(terminal.Green, "Success!") // Spinner for long operations spinner := terminal.NewSpinner("Loading...") spinner.Start() // ... do work ... spinner.Stop() ``` ### 3. Access Configuration ```go theme={null} import "github.com/osmedeus/osmedeus-ng/internal/config" func runMyCommand() error { // Load config (uses global settingsFile from root.go) cfg, err := config.Load(settingsFile) if err != nil { return err } // Use config values baseFolder := cfg.BaseFolder dbPath := cfg.Database.DBPath // Access environment paths workflowsPath := cfg.Environments.Workflows binariesPath := cfg.Environments.Binaries return nil } ``` ### 4. Handle Errors ```go theme={null} func runMyCommand() error { // Return errors - Cobra handles display if targetFlag == "" { return fmt.Errorf("--target is required") } result, err := doSomething(targetFlag) if err != nil { return fmt.Errorf("operation failed: %w", err) } return nil } ``` ### 5. Add Command Help ```go theme={null} var myCmd = &cobra.Command{ Use: "mycommand ", Short: "Brief one-line description", Long: `Detailed description of the command. This command does X, Y, and Z. Use it when you need to... Examples: # Basic usage osmedeus mycommand action -t target # With options osmedeus mycommand action -t target -o output.txt --verbose # Multiple targets osmedeus mycommand action -t target1 -t target2`, RunE: func(cmd *cobra.Command, args []string) error { return cmd.Help() }, } ``` ## Example: Export Command ```go theme={null} // pkg/cli/export.go package cli import ( "fmt" "os" "github.com/osmedeus/osmedeus-ng/internal/config" "github.com/osmedeus/osmedeus-ng/internal/database" "github.com/osmedeus/osmedeus-ng/internal/terminal" "github.com/spf13/cobra" ) var ( exportWorkspace string exportFormat string exportOutput string ) var exportCmd = &cobra.Command{ Use: "export", Short: "Export data from workspace", Long: `Export assets, vulnerabilities, or other data from a workspace. Examples: osmedeus export assets -w example.com -f json -o assets.json osmedeus export vulns -w example.com -f csv`, } var exportAssetsCmd = &cobra.Command{ Use: "assets", Short: "Export assets", RunE: func(cmd *cobra.Command, args []string) error { return runExportAssets() }, } var exportVulnsCmd = &cobra.Command{ Use: "vulns", Short: "Export vulnerabilities", RunE: func(cmd *cobra.Command, args []string) error { return runExportVulns() }, } func init() { rootCmd.AddCommand(exportCmd) exportCmd.PersistentFlags().StringVarP(&exportWorkspace, "workspace", "w", "", "Workspace name (required)") exportCmd.PersistentFlags().StringVarP(&exportFormat, "format", "f", "json", "Output format (json, csv, jsonl)") exportCmd.PersistentFlags().StringVarP(&exportOutput, "output", "o", "", "Output file (stdout if empty)") exportCmd.MarkPersistentFlagRequired("workspace") exportCmd.AddCommand(exportAssetsCmd) exportCmd.AddCommand(exportVulnsCmd) } func runExportAssets() error { cfg, err := config.Load(settingsFile) if err != nil { return err } db, err := database.Connect(cfg) if err != nil { return fmt.Errorf("database connection failed: %w", err) } defer db.Close() assets, err := db.GetAssetsByWorkspace(exportWorkspace) if err != nil { return err } output, err := formatOutput(assets, exportFormat) if err != nil { return err } if exportOutput != "" { return os.WriteFile(exportOutput, []byte(output), 0644) } fmt.Println(output) return nil } ``` ## Flag Types ```go theme={null} // String flag cmd.Flags().StringVarP(&myString, "string", "s", "default", "description") // Bool flag cmd.Flags().BoolVarP(&myBool, "bool", "b", false, "description") // Int flag cmd.Flags().IntVarP(&myInt, "count", "c", 10, "description") // String slice (repeatable) cmd.Flags().StringSliceVarP(&mySlice, "item", "i", []string{}, "description") // Persistent flag (inherited by subcommands) cmd.PersistentFlags().StringVarP(&myPersistent, "global", "g", "", "description") // Required flag cmd.Flags().StringVarP(&required, "required", "r", "", "description") cmd.MarkFlagRequired("required") ``` ## Best Practices 1. **Use subcommands** for related actions 2. **Provide aliases** for common commands 3. **Write helpful examples** in Long description 4. **Validate input early** before processing 5. **Use terminal helpers** for consistent output 6. **Handle interrupts** with context ## Next Steps * [Adding API Endpoints](api-endpoints.md) - REST endpoints * [Adding Step Types](step-types.md) - Custom steps * [CLI Reference](../cli/run.md) - Existing commands # Adding Functions Source: https://docs.osmedeus.org/extending/functions Register custom utility functions for use in workflows. ## Overview Functions are implemented in Go and exposed to the Goja JavaScript VM runtime in `internal/functions/`. ## Function Registry Functions are registered in `internal/functions/goja_runtime.go`: ```go theme={null} func (r *GojaRuntime) registerFunctions() { // File functions r.vm.Set("fileExists", r.fileExists) r.vm.Set("fileLength", r.fileLength) r.vm.Set("readFile", r.readFile) // String functions r.vm.Set("trim", r.trim) r.vm.Set("split", r.split) // Add your function r.vm.Set("myNewFunction", r.myNewFunction) } ``` ## Steps to Add a Function ### 1. Implement the Function Add to an appropriate file in `internal/functions/`: ```go theme={null} // internal/functions/util_functions.go func (r *GojaRuntime) myNewFunction(call goja.FunctionCall) goja.Value { // Get arguments if len(call.ArgumentList) < 1 { return goja.Undefined() } arg1 := call.Argument(0).String() // Optional second argument with default arg2 := "default" if len(call.ArgumentList) > 1 { arg2 = call.Argument(1).String() } // Perform operation result, err := doSomething(arg1, arg2) if err != nil { // Return undefined or error value r.vm.Set("_error", err.Error()) return goja.Undefined() } // Convert result to Goja value value, _ := r.vm.ToValue(result) return value } ``` ### 2. Register the Function Update `internal/functions/goja_runtime.go`: ```go theme={null} func (r *GojaRuntime) registerFunctions() { // ... existing registrations ... // Register your new function r.vm.Set("myNewFunction", r.myNewFunction) } ``` ### 3. Add to Function List Update `internal/functions/registry.go` for `ListFunctions()`: ```go theme={null} func (r *Registry) ListFunctions() []FunctionInfo { return []FunctionInfo{ // ... existing functions ... { Name: "myNewFunction", Description: "Description of what it does", Signature: "myNewFunction(arg1, arg2?)", Category: "util", }, } } ``` ### 4. Write Tests Add to `internal/functions/registry_test.go`: ```go theme={null} func TestMyNewFunction(t *testing.T) { registry := functions.NewRegistry() result, err := registry.Execute(`myNewFunction("input", "option")`, nil) require.NoError(t, err) assert.Equal(t, "expected-output", result) } func TestMyNewFunction_DefaultArg(t *testing.T) { registry := functions.NewRegistry() result, err := registry.Execute(`myNewFunction("input")`, nil) require.NoError(t, err) assert.Equal(t, "expected-with-default", result) } ``` ## Return Types ### String ```go theme={null} func (r *GojaRuntime) myStringFunc(call goja.FunctionCall) goja.Value { result := "hello world" value, _ := r.vm.ToValue(result) return value } ``` ### Boolean ```go theme={null} func (r *GojaRuntime) myBoolFunc(call goja.FunctionCall) goja.Value { result := true value, _ := r.vm.ToValue(result) return value } ``` ### Number ```go theme={null} func (r *GojaRuntime) myNumberFunc(call goja.FunctionCall) goja.Value { result := 42 value, _ := r.vm.ToValue(result) return value } ``` ### Array ```go theme={null} func (r *GojaRuntime) myArrayFunc(call goja.FunctionCall) goja.Value { result := []string{"a", "b", "c"} value, _ := r.vm.ToValue(result) return value } ``` ### Object/Map ```go theme={null} func (r *GojaRuntime) myObjectFunc(call goja.FunctionCall) goja.Value { result := map[string]interface{}{ "key1": "value1", "key2": 42, } value, _ := r.vm.ToValue(result) return value } ``` ## Working with Context Functions can access the execution context: ```go theme={null} func (r *GojaRuntime) myContextFunc(call goja.FunctionCall) goja.Value { // Get context variables ctxValue, err := r.vm.Get("_context") if err != nil { return goja.Undefined() } ctx, _ := ctxValue.Export() contextMap := ctx.(map[string]interface{}) target := contextMap["target"].(string) // Use context in function logic result := processWithContext(target) value, _ := r.vm.ToValue(result) return value } ``` ## Example: Hash Function ```go theme={null} // internal/functions/util_functions.go import ( "crypto/md5" "crypto/sha256" "encoding/hex" ) func (r *GojaRuntime) hash(call goja.FunctionCall) goja.Value { if len(call.ArgumentList) < 2 { return goja.Undefined() } input := call.Argument(0).String() algorithm := call.Argument(1).String() var result string switch algorithm { case "md5": hash := md5.Sum([]byte(input)) result = hex.EncodeToString(hash[:]) case "sha256": hash := sha256.Sum256([]byte(input)) result = hex.EncodeToString(hash[:]) default: return goja.Undefined() } value, _ := r.vm.ToValue(result) return value } ``` Register: ```go theme={null} r.vm.Set("hash", r.hash) ``` Usage in workflow: ```yaml theme={null} - name: compute-hash type: function function: hash("{{target}}", "sha256") exports: target_hash: "{{result}}" ``` ## Example: HTTP Fetch Function ```go theme={null} // internal/functions/http_functions.go func (r *GojaRuntime) httpFetch(call goja.FunctionCall) goja.Value { if len(call.ArgumentList) < 1 { return goja.Undefined() } url := call.Argument(0).String() // Optional method (default GET) method := "GET" if len(call.ArgumentList) > 1 { method = call.Argument(1).String() } // Optional headers headers := make(map[string]string) if len(call.ArgumentList) > 2 { headersArg, _ := call.Argument(2).Export() if h, ok := headersArg.(map[string]interface{}); ok { for k, v := range h { headers[k] = fmt.Sprintf("%v", v) } } } // Make request req, err := http.NewRequest(method, url, nil) if err != nil { return goja.Undefined() } for k, v := range headers { req.Header.Set(k, v) } client := &http.Client{Timeout: 30 * time.Second} resp, err := client.Do(req) if err != nil { return goja.Undefined() } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) result := map[string]interface{}{ "status": resp.StatusCode, "body": string(body), "headers": resp.Header, } value, _ := r.vm.ToValue(result) return value } ``` Usage: ```yaml theme={null} - name: fetch-api type: function function: httpFetch("https://api.example.com/data", "GET", {"Authorization": "Bearer token"}) exports: api_data: "{{result.body}}" ``` ## Best Practices 1. **Validate arguments** - Check argument count and types 2. **Handle errors gracefully** - Return undefined for errors 3. **Document the function** - Add to ListFunctions() 4. **Write tests** - Cover happy path and edge cases 5. **Keep functions pure** - Minimize side effects 6. **Support optional arguments** - Use sensible defaults ## Function Categories Organize functions by category: | Category | File | Functions | | -------- | --------------------- | ------------------------------------------- | | `file` | `file_functions.go` | fileExists, fileLength, readFile, writeFile | | `string` | `string_functions.go` | trim, split, join, replace, contains | | `util` | `util_functions.go` | log\_info, getEnvVar, exit | | `http` | `http_functions.go` | http\_get, http\_post | | `db` | `db_functions.go` | db\_select, db\_select\_assets | | `jq` | `jq.go` | jq | ## Next Steps * [Adding Step Types](step-types.md) - Custom step executors * [Functions Reference](../functions/reference.md) - All functions * [Functions Overview](../functions/overview.md) - Usage guide # Adding Runners Source: https://docs.osmedeus.org/extending/runners Add custom execution environments for workflow commands. ## Overview Runners execute bash commands in different environments. Osmedeus includes Host, Docker, and SSH runners. ## Runner Interface All runners implement this interface in `internal/runner/runner.go`: ```go theme={null} type Runner interface { Execute(ctx context.Context, command string) (*CommandResult, error) Setup(ctx context.Context) error Cleanup(ctx context.Context) error Type() core.RunnerType IsRemote() bool } type CommandResult struct { Output string ExitCode int Error error } ``` ## Steps to Add a New Runner ### 1. Define the Type Constant Add to `internal/core/types.go`: ```go theme={null} type RunnerType string const ( RunnerTypeHost RunnerType = "host" RunnerTypeDocker RunnerType = "docker" RunnerTypeSSH RunnerType = "ssh" RunnerTypeMyNew RunnerType = "mynew" // Add your type ) ``` ### 2. Add Configuration (if needed) Update `internal/core/workflow.go`: ```go theme={null} type RunnerConfig struct { // Existing fields... // MyNew runner specific MyNewOption1 string `yaml:"mynew_option1,omitempty"` MyNewOption2 int `yaml:"mynew_option2,omitempty"` } ``` ### 3. Create the Runner Create `internal/runner/mynew_runner.go`: ```go theme={null} package runner import ( "context" "fmt" "github.com/osmedeus/osmedeus-ng/internal/core" ) type MyNewRunner struct { config *core.RunnerConfig // Add any connection/state fields client *SomeClient } func NewMyNewRunner(config *core.RunnerConfig) (*MyNewRunner, error) { if config.MyNewOption1 == "" { return nil, fmt.Errorf("mynew_option1 is required") } return &MyNewRunner{ config: config, }, nil } func (r *MyNewRunner) Setup(ctx context.Context) error { // Initialize connection, create resources, etc. client, err := connectToService(r.config.MyNewOption1) if err != nil { return fmt.Errorf("failed to connect: %w", err) } r.client = client return nil } func (r *MyNewRunner) Execute(ctx context.Context, command string) (*CommandResult, error) { // Check for context cancellation select { case <-ctx.Done(): return nil, ctx.Err() default: } // Execute command in your environment output, exitCode, err := r.client.RunCommand(ctx, command) if err != nil { return &CommandResult{ Output: output, ExitCode: exitCode, Error: err, }, nil } return &CommandResult{ Output: output, ExitCode: exitCode, }, nil } func (r *MyNewRunner) Cleanup(ctx context.Context) error { // Clean up resources if r.client != nil { return r.client.Close() } return nil } func (r *MyNewRunner) Type() core.RunnerType { return core.RunnerTypeMyNew } func (r *MyNewRunner) IsRemote() bool { return true // or false for local execution } // Optional: implement CopyFromRemote for remote runners func (r *MyNewRunner) CopyFromRemote(ctx context.Context, remotePath, localPath string) error { return r.client.DownloadFile(ctx, remotePath, localPath) } ``` ### 4. Register in Factory Update `internal/runner/runner.go`: ```go theme={null} func NewRunnerFromType( runnerType core.RunnerType, config *core.RunnerConfig, binaryPath string, ) (Runner, error) { switch runnerType { case core.RunnerTypeHost: return NewHostRunner() case core.RunnerTypeDocker: return NewDockerRunner(config) case core.RunnerTypeSSH: return NewSSHRunner(config, binaryPath) case core.RunnerTypeMyNew: return NewMyNewRunner(config) default: return nil, fmt.Errorf("unknown runner type: %s", runnerType) } } ``` ### 5. Add Validation Update `internal/parser/validator.go`: ```go theme={null} func (v *Validator) validateRunnerConfig(runnerType core.RunnerType, config *core.RunnerConfig) error { switch runnerType { case core.RunnerTypeMyNew: if config == nil { return fmt.Errorf("runner_config required for mynew runner") } if config.MyNewOption1 == "" { return fmt.Errorf("mynew_option1 is required") } } return nil } ``` ### 6. Write Tests Create `internal/runner/mynew_runner_test.go`: ```go theme={null} package runner import ( "context" "testing" "github.com/osmedeus/osmedeus-ng/internal/core" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) func TestMyNewRunner_Execute(t *testing.T) { config := &core.RunnerConfig{ MyNewOption1: "test-value", } runner, err := NewMyNewRunner(config) require.NoError(t, err) ctx := context.Background() err = runner.Setup(ctx) require.NoError(t, err) defer runner.Cleanup(ctx) result, err := runner.Execute(ctx, "echo hello") require.NoError(t, err) assert.Contains(t, result.Output, "hello") assert.Equal(t, 0, result.ExitCode) } ``` ## Example: Kubernetes Runner ```go theme={null} // internal/runner/k8s_runner.go type K8sRunner struct { config *core.RunnerConfig clientset *kubernetes.Clientset pod *v1.Pod } func NewK8sRunner(config *core.RunnerConfig) (*K8sRunner, error) { kubeconfig, err := clientcmd.BuildConfigFromFlags("", config.KubeConfigPath) if err != nil { return nil, err } clientset, err := kubernetes.NewForConfig(kubeconfig) if err != nil { return nil, err } return &K8sRunner{ config: config, clientset: clientset, }, nil } func (r *K8sRunner) Setup(ctx context.Context) error { // Create pod for execution pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: fmt.Sprintf("osmedeus-%s", uuid.New().String()[:8]), Namespace: r.config.Namespace, }, Spec: v1.PodSpec{ Containers: []v1.Container{{ Name: "runner", Image: r.config.Image, Command: []string{"sleep", "infinity"}, }}, RestartPolicy: v1.RestartPolicyNever, }, } created, err := r.clientset.CoreV1().Pods(r.config.Namespace).Create(ctx, pod, metav1.CreateOptions{}) if err != nil { return err } r.pod = created // Wait for pod to be ready return r.waitForPod(ctx) } func (r *K8sRunner) Execute(ctx context.Context, command string) (*CommandResult, error) { req := r.clientset.CoreV1().RESTClient().Post(). Resource("pods"). Name(r.pod.Name). Namespace(r.config.Namespace). SubResource("exec"). VersionedParams(&v1.PodExecOptions{ Container: "runner", Command: []string{"sh", "-c", command}, Stdout: true, Stderr: true, }, scheme.ParameterCodec) exec, err := remotecommand.NewSPDYExecutor(r.config, "POST", req.URL()) if err != nil { return nil, err } var stdout, stderr bytes.Buffer err = exec.StreamWithContext(ctx, remotecommand.StreamOptions{ Stdout: &stdout, Stderr: &stderr, }) return &CommandResult{ Output: stdout.String() + stderr.String(), // Extract exit code from error if available }, err } func (r *K8sRunner) Cleanup(ctx context.Context) error { if r.pod != nil { return r.clientset.CoreV1().Pods(r.config.Namespace).Delete(ctx, r.pod.Name, metav1.DeleteOptions{}) } return nil } ``` Usage in workflow: ```yaml theme={null} kind: module name: k8s-scan runner: k8s runner_config: kubeconfig_path: ~/.kube/config namespace: osmedeus image: alpine:latest steps: - name: scan type: bash command: nmap -sV {{target}} ``` ## Best Practices 1. **Implement context cancellation** - Check `ctx.Done()` for graceful shutdown 2. **Clean up resources** - Always cleanup in `Cleanup()` method 3. **Handle reconnection** - For remote runners, handle connection drops 4. **Log operations** - Use structured logging for debugging 5. **Validate configuration** - Check required fields early 6. **Support file transfer** - Implement `CopyFromRemote` if applicable ## Next Steps * [Adding Step Types](step-types.md) - Custom step executors * [Adding Functions](functions.md) - Utility functions * [Runners Concept](../concepts/runners.md) - Runner overview # Adding Step Types Source: https://docs.osmedeus.org/extending/step-types Add custom step executors to extend workflow capabilities. ## Overview Step types are handlers that execute specific step configurations. Each type has a dedicated executor in `internal/executor/`. ## Steps to Add a New Step Type ### 1. Define the Type Constant Add to `internal/core/types.go`: ```go theme={null} // StepType represents the type of step type StepType string const ( StepTypeBash StepType = "bash" StepTypeFunction StepType = "function" StepTypeForeach StepType = "foreach" StepTypeParallelSteps StepType = "parallel-steps" StepTypeRemoteBash StepType = "remote-bash" StepTypeHTTP StepType = "http" StepTypeLLM StepType = "llm" StepTypeMyNew StepType = "mynew" // Add your new type ) ``` ### 2. Add Step Fields (if needed) Add fields to `internal/core/step.go`: ```go theme={null} type Step struct { // Existing fields... // New fields for your step type MyNewField string `yaml:"my_new_field,omitempty"` MyNewConfig *MyNewConfig `yaml:"my_new_config,omitempty"` } type MyNewConfig struct { Option1 string `yaml:"option1,omitempty"` Option2 int `yaml:"option2,omitempty"` } ``` ### 3. Create the Executor Create `internal/executor/mynew_executor.go`: ```go theme={null} package executor import ( "context" "github.com/osmedeus/osmedeus-ng/internal/core" "github.com/osmedeus/osmedeus-ng/internal/template" ) type MyNewExecutor struct { templateEngine *template.Engine } func NewMyNewExecutor(templateEngine *template.Engine) *MyNewExecutor { return &MyNewExecutor{ templateEngine: templateEngine, } } func (e *MyNewExecutor) Execute( ctx context.Context, step *core.Step, execCtx *core.ExecutionContext, ) (*core.StepResult, error) { // 1. Render templates in step fields renderedField, err := e.templateEngine.Render(step.MyNewField, execCtx.Variables) if err != nil { return nil, fmt.Errorf("template render failed: %w", err) } // 2. Perform your step logic output, err := e.doSomething(ctx, renderedField, step.MyNewConfig) if err != nil { return &core.StepResult{ Success: false, Error: err, }, nil } // 3. Return result return &core.StepResult{ Success: true, Output: output, }, nil } func (e *MyNewExecutor) doSomething( ctx context.Context, field string, config *core.MyNewConfig, ) (string, error) { // Implementation here return "result", nil } ``` ### 4. Register in Dispatcher Update `internal/executor/dispatcher.go`: ```go theme={null} type StepDispatcher struct { bashExecutor *BashExecutor functionExecutor *FunctionExecutor foreachExecutor *ForeachExecutor parallelExecutor *ParallelExecutor remoteBashExecutor *RemoteBashExecutor httpExecutor *HTTPExecutor llmExecutor *LLMExecutor myNewExecutor *MyNewExecutor // Add your executor runner runner.Runner } func NewStepDispatcher( templateEngine *template.Engine, functionRegistry *functions.Registry, runner runner.Runner, ) *StepDispatcher { return &StepDispatcher{ // Existing executors... myNewExecutor: NewMyNewExecutor(templateEngine), runner: runner, } } func (d *StepDispatcher) Dispatch( ctx context.Context, step *core.Step, execCtx *core.ExecutionContext, ) (*core.StepResult, error) { switch step.Type { case core.StepTypeBash: return d.bashExecutor.Execute(ctx, step, execCtx, d.runner) // ... other cases ... case core.StepTypeMyNew: return d.myNewExecutor.Execute(ctx, step, execCtx) default: return nil, fmt.Errorf("unknown step type: %s", step.Type) } } ``` ### 5. Add Validation Update `internal/parser/validator.go`: ```go theme={null} func (v *Validator) validateStep(step *core.Step) error { switch step.Type { // ... existing cases ... case core.StepTypeMyNew: return v.validateMyNewStep(step) } return nil } func (v *Validator) validateMyNewStep(step *core.Step) error { if step.MyNewField == "" { return fmt.Errorf("step '%s': my_new_field is required for mynew step type", step.Name) } return nil } ``` ### 6. Write Tests Create `internal/executor/mynew_executor_test.go`: ```go theme={null} package executor import ( "context" "testing" "github.com/osmedeus/osmedeus-ng/internal/core" "github.com/osmedeus/osmedeus-ng/internal/template" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) func TestMyNewExecutor_Execute(t *testing.T) { engine := template.NewEngine() executor := NewMyNewExecutor(engine) step := &core.Step{ Name: "test-step", Type: core.StepTypeMyNew, MyNewField: "test-value", } execCtx := &core.ExecutionContext{ Variables: map[string]interface{}{ "target": "example.com", }, } result, err := executor.Execute(context.Background(), step, execCtx) require.NoError(t, err) assert.True(t, result.Success) assert.NotEmpty(t, result.Output) } ``` ## Example: Custom Notification Step ```go theme={null} // internal/executor/notify_executor.go type NotifyExecutor struct { templateEngine *template.Engine } func (e *NotifyExecutor) Execute( ctx context.Context, step *core.Step, execCtx *core.ExecutionContext, ) (*core.StepResult, error) { // Render message template message, err := e.templateEngine.Render(step.NotifyMessage, execCtx.Variables) if err != nil { return nil, err } // Send notification based on channel switch step.NotifyChannel { case "slack": err = e.sendSlack(ctx, step.NotifyConfig.WebhookURL, message) case "discord": err = e.sendDiscord(ctx, step.NotifyConfig.WebhookURL, message) case "telegram": err = e.sendTelegram(ctx, step.NotifyConfig, message) } if err != nil { return &core.StepResult{Success: false, Error: err}, nil } return &core.StepResult{Success: true, Output: "Notification sent"}, nil } ``` Usage in workflow: ```yaml theme={null} - name: notify-complete type: notify notify_channel: slack notify_message: "Scan completed for {{target}}" notify_config: webhook_url: "{{slack_webhook}}" ``` ## Best Practices 1. **Always render templates** before using step fields 2. **Support context cancellation** via `ctx.Done()` 3. **Return meaningful errors** with context 4. **Export useful values** via StepResult 5. **Write comprehensive tests** 6. **Add validation rules** for required fields ## Next Steps * [Adding Runners](runners.md) - Custom execution environments * [Adding Functions](functions.md) - Utility functions * [Architecture](../concepts/architecture.md) - System overview # Functions Overview Source: https://docs.osmedeus.org/functions/overview Osmedeus provides 190+ utility functions via a Goja JavaScript runtime. Functions can be used in workflow steps, conditions, and evaluated from the CLI or API. ## Usage ### In Steps ```yaml theme={null} - name: log-start type: function function: log_info("Scanning {{target}}") - name: check-file type: function functions: - log_info("Checking files") - file_exists("{{Output}}/data.txt") ``` ### In Conditions ```yaml theme={null} - name: scan type: bash pre_condition: 'file_length("{{Output}}/hosts.txt") > 0' command: nuclei -l {{Output}}/hosts.txt ``` ### In Flow Conditions ```yaml theme={null} modules: - name: vuln-scan path: modules/vuln.yaml condition: 'file_exists("{{Output}}/live.txt")' ``` ## Function Step Types ### Single Function ```yaml theme={null} - name: log type: function function: log_info("Message") ``` ### Multiple Functions (Sequential) ```yaml theme={null} - name: setup type: function functions: - log_info("Step 1") - log_info("Step 2") - log_info("Step 3") ``` ### Parallel Functions ```yaml theme={null} - name: parallel-checks type: function parallel_functions: - file_length("{{Output}}/file1.txt") - file_length("{{Output}}/file2.txt") - file_length("{{Output}}/file3.txt") ``` ## Return Values Functions return values that can be: ### Used in Exports ```yaml theme={null} - name: count-lines type: function function: file_length("{{Output}}/hosts.txt") exports: host_count: "{{result}}" ``` ### Used in Conditions ```yaml theme={null} - name: scan type: bash pre_condition: 'file_length("{{Output}}/hosts.txt") > 0' command: scan {{Output}}/hosts.txt ``` ### Used in Decision Routing ```yaml theme={null} - name: check type: function function: file_exists("{{Output}}/critical.txt") exports: has_critical: "{{result}}" decision: switch: "{{has_critical}}" cases: "true": { goto: handle-critical } default: { goto: continue-normal } ``` ## CLI Evaluation ### Basic Evaluation ```bash theme={null} # List all functions osmedeus func list # Evaluate a function osmedeus func e 'file_exists("/path/to/file")' # With target variable osmedeus func e 'log_info("Scanning " + target)' -t example.com # With custom parameters osmedeus func e 'log_info(prefix + target)' -t example.com --params 'prefix=test_' ``` ### Script Source Priority The CLI determines the script to execute in this order: 1. `--function-file` - Read script from file 2. `-f/--function` - Function name with remaining args as arguments 3. Positional argument - Direct expression after `e` or `eval` 4. `-e/--eval` - Script via flag 5. `--stdin` - Read from stdin ### Bulk Processing Process multiple targets from a file: ```bash theme={null} # Process targets from file osmedeus func e 'log_info("Processing: " + target)' -T targets.txt # With concurrency osmedeus func e 'http_get("https://" + target)' -T targets.txt -c 10 # Using function files osmedeus func e --function-file check.js -T targets.txt -c 5 # With parameters osmedeus func e 'log_info(prefix + target)' -T targets.txt --params 'prefix=test_' -c 5 ``` ### Function List Options ```bash theme={null} # List all functions osmedeus func list # Search/filter functions osmedeus func list -s "event" osmedeus func list -s "file" # Show examples osmedeus func list --example # Custom column width osmedeus func list --width 80 ``` ## API Evaluation Evaluate functions via REST API: ```bash theme={null} curl -X POST http://localhost:8002/osm/api/functions/eval \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"script": "file_length(\"/path/to/file\")"}' ``` List available functions: ```bash theme={null} curl http://localhost:8002/osm/api/functions/list \ -H "Authorization: Bearer $TOKEN" ``` ## Context Variables Functions have access to execution context: ```javascript theme={null} // Built-in variables are available log_info("Target: " + target) log_info("Output: " + "{{Output}}") // Exports from previous steps log_info("Previous result: " + "{{previous_export}}") ``` ## Error Handling Functions that fail don't stop workflow execution unless you configure error handling: ```yaml theme={null} - name: risky-function type: function function: read_file("/possibly/missing/file.txt") on_error: - action: log message: "Function failed" - action: continue ``` ## Function Categories Osmedeus provides 190+ functions organized into 34 categories: | Category | Description | Count | | --------------------- | ----------------------------------------- | ----- | | **File** | File/directory operations, grep, glob | 22 | | **String** | String manipulation, regex matching | 21 | | **Type Conversion** | Parse/convert between types | 4 | | **Type Detection** | Detect input types (file, url, ip, etc.) | 7 | | **Utility** | General utilities (len, exec, sleep) | 11 | | **Logging** | Log messages with level prefixes | 4 | | **Color Printing** | Colored terminal output | 4 | | **Runtime Variables** | Get/set runtime variables | 2 | | **HTTP** | HTTP requests and IP resolution | 4 | | **LLM** | LLM interaction and conversations | 3 | | **Generation** | Random strings and UUIDs | 2 | | **Encoding** | Base64 encode/decode | 2 | | **Data Query** | JQ-style JSON querying | 2 | | **Notification** | Telegram and webhook notifications | 8 | | **Event Generation** | Structured event generation | 2 | | **CDN/Storage** | Cloud storage operations (S3-compatible) | 11 | | **Unix Commands** | Wrappers for sort, wget, git, tar, etc. | 11 | | **Archive (Go)** | Pure Go zip/unzip implementations | 3 | | **Snapshot** | Workspace export/import as ZIP archives | 2 | | **Diff** | File comparison and diff extraction | 1 | | **Output** | Save content, JSONL/CSV conversion | 6 | | **URL Processing** | URL deduplication, filtering, and parsing | 6 | | **Markdown** | Markdown rendering and conversion | 6 | | **Database** | Asset/vuln import, queries, stats | 50 | | **SARIF** | SARIF parsing and database import | 2 | | **Nmap/Port** | Port scanning and result import | 3 | | **Installer** | Download packages via go-getter/Nix | 4 | | **Environment** | Environment variable operations | 2 | | **Tmux** | Background process management via tmux | 5 | | **SSH/Sync** | Remote execution and file sync | 5 | | **Script Execution** | Python and TypeScript execution | 4 | | **Agent/Distributed** | ACP agents and distributed execution | 3 | | **Module Control** | Skip module and run modules/flows | 3 | | **Authentication** | Sudo authentication | 1 | ## Best Practices 1. **Use functions for conditions** ```yaml theme={null} pre_condition: 'file_exists("{{Output}}/input.txt")' ``` 2. **Log meaningful messages** ```yaml theme={null} function: log_info("Found " + host_count + " hosts for {{Target}}") ``` 3. **Export function results** ```yaml theme={null} exports: line_count: "{{result}}" ``` 4. **Handle missing files gracefully** ```yaml theme={null} pre_condition: 'file_exists("{{Output}}/data.txt")' ``` 5. **Use appropriate logging levels** * `log_debug` for verbose debugging * `log_info` for informational messages * `log_warn` for warnings * `log_error` for errors 6. **Leverage bulk processing for testing** ```bash theme={null} # Test a function against many targets osmedeus func e 'http_get("https://" + target + "/api/health")' -T domains.txt -c 20 ``` ## Next Steps * [Functions Reference](reference) - Complete function list with signatures * [Control Flow](/workflows/control-flow) - Using conditions and decisions * [Variables](/workflows/variables) - Exports and parameters # Functions Reference Source: https://docs.osmedeus.org/functions/reference Complete reference for all 190+ utility functions organized by category. ## File Functions Operations on files and directories. ### file\_exists(path) Check if a file exists. ```javascript theme={null} file_exists("/path/to/file.txt") // Returns: true or false ``` ### file\_length(path) Count non-empty lines in a file. ```javascript theme={null} file_length("{{Output}}/hosts.txt") // Returns: 42 ``` ### dir\_length(path) Count entries in a directory. ```javascript theme={null} dir_length("{{Output}}/screenshots") // Returns: 15 ``` ### file\_contains(path, pattern) Check if file contains a pattern. ```javascript theme={null} file_contains("{{Output}}/urls.txt", "admin") // Returns: true or false ``` ### regex\_extract(path, pattern) Extract lines matching a regex from file. ```javascript theme={null} regex_extract("{{Output}}/urls.txt", ".*api.*") // Returns: ["https://api.example.com", ...] ``` ### read\_file(path) Read entire file contents. ```javascript theme={null} read_file("{{Output}}/config.json") // Returns: file contents as string ``` ### read\_lines(path) Read file as array of lines. ```javascript theme={null} read_lines("{{Output}}/subdomains.txt") // Returns: ["sub1.example.com", "sub2.example.com", ...] ``` ### remove\_file(path) Delete a file. ```javascript theme={null} remove_file("{{Output}}/temp.txt") // Returns: true or false ``` ### remove\_folder(path) Delete folder recursively. ```javascript theme={null} remove_folder("{{Output}}/cache") // Returns: true or false ``` ### rm\_rf(path) Delete file or folder recursively (like `rm -rf`). ```javascript theme={null} rm_rf("{{Output}}/tmp") // Returns: true or false ``` ### remove\_all\_except(folder, keep\_file) Remove everything under folder except the specified file. ```javascript theme={null} remove_all_except("{{Output}}", "{{Output}}/keep.txt") // Returns: true or false ``` ### create\_folder(path) Create folder recursively (like `mkdir -p`). ```javascript theme={null} create_folder("{{Output}}/new-folder") // Returns: true or false ``` ### append\_file(dest, source) Append source file content to destination file. ```javascript theme={null} append_file("{{Output}}/all.txt", "{{Output}}/part.txt") // Returns: true or false ``` ### move\_file(source, dest) Move file from source to destination. ```javascript theme={null} move_file("{{Output}}/raw.txt", "{{Output}}/processed.txt") // Returns: true or false ``` ### glob(pattern) List filenames matching glob pattern. ```javascript theme={null} glob("{{Output}}/*.txt") // Returns: ["file1.txt", "file2.txt", ...] ``` ### grep\_string\_to\_file(dest, source, str) Write lines containing string to destination file. ```javascript theme={null} grep_string_to_file("{{Output}}/admin-urls.txt", "{{Output}}/urls.txt", "admin") // Returns: true or false ``` ### grep\_regex\_to\_file(dest, source, pattern) Write lines matching regex to destination file. ```javascript theme={null} grep_regex_to_file("{{Output}}/api-urls.txt", "{{Output}}/urls.txt", ".*api.*") // Returns: true or false ``` ### grep\_string(source, str) Return lines containing string. ```javascript theme={null} grep_string("{{Output}}/urls.txt", "admin") // Returns: "https://example.com/admin\nhttps://example.com/admin/login" ``` ### grep\_regex(source, pattern) Return lines matching regex. ```javascript theme={null} grep_regex("{{Output}}/urls.txt", ".*api.*") // Returns: matching lines as string ``` ### remove\_blank\_lines(path) Remove blank lines from file in-place. ```javascript theme={null} remove_blank_lines("{{Output}}/urls.txt") // Returns: true or false ``` ### chunk\_file(input, lines\_per\_chunk, output) Split a file into chunks of N lines each, writing numbered output files. ```javascript theme={null} chunk_file("{{Output}}/urls.txt", 1000, "{{Output}}/chunks/urls") // Creates: urls-0.txt, urls-1.txt, ... // Returns: true or false ``` ### cut\_to\_file(input\_file, delim, field, output\_file) Extract a specific field from each line using a delimiter and write results to a file. ```javascript theme={null} cut_to_file("{{Output}}/data.csv", ",", 2, "{{Output}}/column2.txt") // Returns: true or false ``` ## String Functions String manipulation operations. ### trim(str) Remove leading/trailing whitespace. ```javascript theme={null} trim(" hello world ") // Returns: "hello world" ``` ### split(str, delim) Split string by delimiter into array. ```javascript theme={null} split("a,b,c", ",") // Returns: ["a", "b", "c"] ``` ### join(arr, delim) Join array elements with delimiter. ```javascript theme={null} join(["a", "b", "c"], "-") // Returns: "a-b-c" ``` ### replace(str, old, new) Replace all occurrences of old with new. ```javascript theme={null} replace("hello world", "world", "there") // Returns: "hello there" ``` ### contains(str, substr) Check if string contains substring. ```javascript theme={null} contains("hello world", "world") // Returns: true ``` ### starts\_with(str, prefix) Check if string starts with prefix. ```javascript theme={null} starts_with("hello", "hel") // Returns: true ``` ### ends\_with(str, suffix) Check if string ends with suffix. ```javascript theme={null} ends_with("hello.txt", ".txt") // Returns: true ``` ### to\_lower\_case(str) Convert to lowercase. ```javascript theme={null} to_lower_case("HELLO") // Returns: "hello" ``` ### to\_upper\_case(str) Convert to uppercase. ```javascript theme={null} to_upper_case("hello") // Returns: "HELLO" ``` ### match(str, pattern) Check if string matches regex pattern. ```javascript theme={null} match("test123", "[0-9]+") // Returns: true ``` ### regex\_match(pattern, str) Check if string matches regex (pattern first). ```javascript theme={null} regex_match("[0-9]+", "test123") // Returns: true ``` ### cut\_with\_delim(input, delim, field) Extract field by delimiter (1-indexed, like `cut`). ```javascript theme={null} cut_with_delim("a:b:c", ":", 2) // Returns: "b" ``` ### normalize\_path(input) Replace special characters (/ | : etc.) with underscore. ```javascript theme={null} normalize_path("test/path:file") // Returns: "test_path_file" ``` ### normal\_path(input) Normalize to path-friendly format (same as `{{TargetSpace}}`). ```javascript theme={null} normal_path("https://example.com/path") // Returns: "example.com_path" ``` ### clean\_sub(path, target?) Clean and deduplicate subdomains in file, optionally filter by target domain. ```javascript theme={null} clean_sub("{{Output}}/subdomains.txt", "example.com") // Returns: true or false ``` ### trim\_left(input, substring) Trim a substring from the left/start of a string. ```javascript theme={null} trim_left("https://example.com", "https://") // Returns: "example.com" ``` ### trim\_right(input, substring) Trim a substring from the right/end of a string. ```javascript theme={null} trim_right("example.com/", "/") // Returns: "example.com" ``` ### trim\_string(input, substring) Trim a substring from both ends of a string. ```javascript theme={null} trim_string("---hello---", "---") // Returns: "hello" ``` ### cut\_space(input, field) Split string by whitespace and extract field (1-indexed). ```javascript theme={null} cut_space("hello world foo", 2) // Returns: "world" ``` ### get\_target\_space(input) Sanitize and truncate input for use as a workspace-safe target name (same as `{{TargetSpace}}`). ```javascript theme={null} get_target_space("https://example.com/path") // Returns: "example.com_path" ``` ### pick\_valid(v1, v2, ..., v10) Return the first non-empty value from the given arguments (up to 10). ```javascript theme={null} pick_valid("", "", "fallback") // Returns: "fallback" pick_valid(target, "default.com") // Returns: target if non-empty, else "default.com" ``` **Aliases:** `cut` is an alias for `cut_with_delim`. `bash` is an alias for `exec_cmd`. ## Type Conversion Functions Convert between data types. ### parse\_int(str) Parse string to integer. ```javascript theme={null} parse_int("42") // Returns: 42 ``` ### parse\_float(str) Parse string to float. ```javascript theme={null} parse_float("3.14") // Returns: 3.14 ``` ### to\_string(val) Convert value to string. ```javascript theme={null} to_string(123) // Returns: "123" ``` ### to\_boolean(val) Convert value to boolean. ```javascript theme={null} to_boolean("true") // Returns: true to_boolean(1) // Returns: true ``` ## Type Detection Functions Detect input types. ### get\_types(input) Detect input type: file, folder, cidr, ip, url, domain, or string. ```javascript theme={null} get_types("192.168.1.0/24") // Returns: "cidr" get_types("example.com") // Returns: "domain" get_types("https://example.com") // Returns: "url" get_types("/etc/passwd") // Returns: "file" ``` ### is\_file(path) Check if path is an existing file. ```javascript theme={null} is_file("/tmp/data.txt") // Returns: true or false ``` ### is\_dir(path) Check if path is an existing directory. ```javascript theme={null} is_dir("/tmp/output") // Returns: true or false ``` ### is\_git(path) Check if path is inside a git repository. ```javascript theme={null} is_git("/path/to/project") // Returns: true or false ``` ### is\_url(input) Check if input is a valid URL. ```javascript theme={null} is_url("https://example.com") // Returns: true is_url("not-a-url") // Returns: false ``` ### is\_compress(path) Check if path is a compressed archive file (.zip, .tar.gz, .tgz, etc.). ```javascript theme={null} is_compress("archive.tar.gz") // Returns: true is_compress("file.txt") // Returns: false ``` ### detect\_language(path) Detect the dominant programming language of a source folder (supports 26+ languages). ```javascript theme={null} detect_language("/path/to/project") // Returns: "javascript" detect_language("{{Output}}/repo") // Returns: "python" ``` ## Utility Functions General utility operations. ### len(val) Get length of string or array. ```javascript theme={null} len("hello") // Returns: 5 len([1, 2, 3]) // Returns: 3 ``` ### is\_empty(val) Check if value is empty. ```javascript theme={null} is_empty("") // Returns: true is_empty("hello") // Returns: false ``` ### is\_not\_empty(val) Check if value is not empty. ```javascript theme={null} is_not_empty("test") // Returns: true is_not_empty("") // Returns: false ``` ### printf(message) Print message to stdout. ```javascript theme={null} printf("Scan started for " + target) ``` ### cat\_file(path) Print file contents to stdout. ```javascript theme={null} cat_file("{{Output}}/results.txt") ``` ### exit(code) Exit the scan with specified code. ```javascript theme={null} exit(0) // Success exit(1) // Error ``` ### exec\_cmd(command) Execute bash command and return output. ```javascript theme={null} exec_cmd("whoami") // Returns: "root" exec_cmd("date") // Returns: "Mon Jan 20 10:30:00 UTC 2025" ``` ### sleep(seconds) Pause execution for n seconds. ```javascript theme={null} sleep(5) // Pause for 5 seconds ``` ### command\_exists(command) Check if command exists in PATH. ```javascript theme={null} command_exists("nmap") // Returns: true or false command_exists("nuclei") // Returns: true or false ``` ## Logging Functions Log messages with level prefixes. ### log\_debug(message) Log debug message with `[DEBUG]` prefix. ```javascript theme={null} log_debug("Processing target: " + target) ``` ### log\_info(message) Log info message with `[INFO]` prefix. ```javascript theme={null} log_info("Scan completed successfully") ``` ### log\_warn(message) Log warning message with `[WARN]` prefix. ```javascript theme={null} log_warn("Rate limit approaching") ``` ### log\_error(message) Log error message with `[ERROR]` prefix. ```javascript theme={null} log_error("Failed to connect to target") ``` ## Color Printing Functions Print messages with colored output. ### print\_green(message) Print message in green color. ```javascript theme={null} print_green("Success!") ``` ### print\_blue(message) Print message in blue color. ```javascript theme={null} print_blue("Processing {{Target}}") ``` ### print\_yellow(message) Print message in yellow color. ```javascript theme={null} print_yellow("Warning: Rate limit hit") ``` ### print\_red(message) Print message in red color. ```javascript theme={null} print_red("Error occurred") ``` ## Runtime Variable Functions Set and get variables at runtime. ### set\_var(name, value) Set a runtime variable for later retrieval. ```javascript theme={null} set_var("api_url", "https://api.example.com") ``` ### get\_var(name) Get a runtime variable value. ```javascript theme={null} get_var("api_url") // Returns: "https://api.example.com" ``` ## HTTP Functions HTTP requests and network operations. ### http\_request(url, method, headers, body) Make HTTP request with full control. ```javascript theme={null} http_request("https://api.example.com/data", "POST", {"Authorization": "Bearer token", "Content-Type": "application/json"}, '{"key":"value"}') // Returns: {statusCode: 200, body: "...", headers: {...}} ``` ### http\_get(url) HTTP GET request with structured response. ```javascript theme={null} http_get("https://api.example.com/data") // Returns: {statusCode: 200, body: "...", headers: {...}} ``` ### http\_post(url, body) HTTP POST request with structured response. ```javascript theme={null} http_post("https://api.example.com/submit", '{"key":"value"}') // Returns: {statusCode: 200, body: "...", headers: {...}} ``` ### get\_ip(domain\_or\_url) Resolve domain or URL to IP address. ```javascript theme={null} get_ip("example.com") // Returns: "93.184.216.34" get_ip("https://example.com/path") // Returns: "93.184.216.34" (auto-parses URL) ``` ## Generation Functions Generate random values. ### random\_string(length) Generate random alphanumeric string. ```javascript theme={null} random_string(16) // Returns: "aB3xY9kLm2nP7qRs" ``` ### uuid() Generate UUID v4. ```javascript theme={null} uuid() // Returns: "550e8400-e29b-41d4-a716-446655440000" ``` ## Encoding Functions Encode and decode data. ### base64\_encode(str) Encode string to base64. ```javascript theme={null} base64_encode("hello") // Returns: "aGVsbG8=" ``` ### base64\_decode(str) Decode base64 string. ```javascript theme={null} base64_decode("aGVsbG8=") // Returns: "hello" ``` ## Data Query Functions Query structured data. ### jq(jsonData, query) Extract data using jq syntax. ```javascript theme={null} jq('{"name":"test","version":"1.0"}', '.name') // Returns: "test" jq('{"items":[1,2,3]}', '.items[]') // Returns: [1, 2, 3] jq('{"a":{"b":"c"}}', '.a.b') // Returns: "c" ``` ### jq\_from\_file(path, query) Extract data using jq from JSON file. ```javascript theme={null} jq_from_file("{{Output}}/data.json", ".results[].url") ``` ## Notification Functions Send notifications via various channels. ### notify\_telegram(message) Send message to configured Telegram chat. ```javascript theme={null} notify_telegram("Scan completed for {{Target}}") // Returns: true or false ``` ### send\_telegram\_file(path, caption?) Send file to Telegram with optional caption. ```javascript theme={null} send_telegram_file("{{Output}}/report.pdf", "Scan report for {{Target}}") // Returns: true or false ``` ### notify\_webhook(message) Send message to all configured webhooks. ```javascript theme={null} notify_webhook("Scan completed for {{Target}}") // Returns: true or false ``` ### send\_webhook\_event(eventType, data) Send structured event to all webhooks. ```javascript theme={null} send_webhook_event("scan_complete", {target: "{{Target}}", status: "success"}) // Returns: true or false ``` ### notify\_telegram\_channel(channel, message) Send message to a specific Telegram channel. ```javascript theme={null} notify_telegram_channel("alerts", "Critical finding on {{Target}}") // Returns: true or false ``` ### send\_telegram\_file\_channel(channel, path, caption?) Send file to a specific Telegram channel with optional caption. ```javascript theme={null} send_telegram_file_channel("reports", "{{Output}}/report.pdf", "Report for {{Target}}") // Returns: true or false ``` ### notify\_message\_as\_file\_telegram(path) Send file content as a text file to Telegram. ```javascript theme={null} notify_message_as_file_telegram("{{Output}}/summary.txt") // Returns: true or false ``` ### notify\_message\_as\_file\_telegram\_channel(channel, path) Send file content as a text file to a specific Telegram channel. ```javascript theme={null} notify_message_as_file_telegram_channel("reports", "{{Output}}/summary.txt") // Returns: true or false ``` ## Event Generation Functions Generate structured events for the event system. ### generate\_event(workspace, topic, source, data\_type, data) Generate a structured event and send to server/webhooks. ```javascript theme={null} // Simple string data generate_event("{{Workspace}}", "discovery", "subdomain-scan", "domain", "api.example.com") // Complex object data generate_event("{{Workspace}}", "vulnerability", "nuclei", "finding", { url: "https://example.com/admin", severity: "critical", template: "CVE-2024-1234" }) // Returns: true (always true - events are queued if server unavailable) ``` **Parameters:** * `workspace` - Workspace name (required, use `{{Workspace}}`) * `topic` - Event category (e.g., "discovery", "vulnerability") * `source` - Origin of the event (e.g., "amass", "nuclei") * `data_type` - Type of data (e.g., "domain", "url", "finding") * `data` - The actual data payload (string or object) **Event Payload Structure:** ```json theme={null} { "workspace": "example.com", "topic": "discovery", "source": "amass", "data_type": "subdomain", "data": "api.example.com", "run_id": "abc123", "workflow_name": "recon", "timestamp": "2025-01-15T10:30:00Z" } ``` ### generate\_event\_from\_file(workspace, topic, source, data\_type, path) Read a file and generate an event for each non-empty line. ```javascript theme={null} generate_event_from_file("{{Workspace}}", "discovery", "amass", "subdomain", "{{Output}}/subdomains.txt") // Returns: 42 (count of events generated) ``` **Parameters:** * `workspace` - Workspace name (required) * `topic` - Event category * `source` - Origin of the event * `data_type` - Type of data * `path` - Path to file (one item per line) ## CDN/Storage Functions Cloud storage operations (S3-compatible). ### cdn\_upload(localPath, remotePath) Upload file to cloud storage. ```javascript theme={null} cdn_upload("{{Output}}/report.zip", "scans/{{Target}}/report.zip") // Returns: true or false ``` ### cdn\_download(remotePath, localPath) Download file from cloud storage. ```javascript theme={null} cdn_download("wordlists/common.txt", "/tmp/common.txt") // Returns: true or false ``` ### cdn\_exists(remotePath) Check if file exists in cloud storage. ```javascript theme={null} cdn_exists("scans/{{Target}}/report.zip") // Returns: true or false ``` ### cdn\_delete(remotePath) Delete file from cloud storage. ```javascript theme={null} cdn_delete("scans/{{Target}}/old-report.zip") // Returns: true or false ``` ### cdn\_sync\_upload(localDir, remotePrefix) Sync local directory to cloud storage (delta upload). ```javascript theme={null} cdn_sync_upload("{{Output}}", "scans/{{Target}}/") // Returns: {uploaded: 5, skipped: 10, errors: 0} ``` ### cdn\_sync\_download(remotePrefix, localDir) Sync cloud storage to local directory (delta download). ```javascript theme={null} cdn_sync_download("base-setup/", "{{BaseFolder}}") // Returns: {downloaded: 3, skipped: 7, errors: 0} ``` ### cdn\_get\_presigned\_url(remotePath, expiryMins?) Generate presigned URL for file access. ```javascript theme={null} cdn_get_presigned_url("scans/target/report.zip", 60) // Returns: "https://bucket.s3.amazonaws.com/scans/target/report.zip?X-Amz-..." ``` ### cdn\_list(prefix?) List files with metadata from cloud storage. ```javascript theme={null} cdn_list("scans/") // Returns: [{key: "scans/target1/report.zip", size: 1024, lastModified: "..."}, ...] ``` ### cdn\_stat(remotePath) Get file metadata from cloud storage. ```javascript theme={null} cdn_stat("scans/target/report.zip") // Returns: {key: "...", size: 1024, lastModified: "...", etag: "..."} or null ``` ### cdn\_read(remotePath) Read file content from cloud storage and return as string. ```javascript theme={null} cdn_read("config/settings.yaml") // Returns: file contents as string ``` ### cdn\_ls\_tree(prefix?, depth?) List cloud storage contents in a tree-like format. ```javascript theme={null} cdn_ls_tree("scans/") // Returns: tree-formatted string cdn_ls_tree("scans/", 2) // Returns: tree limited to depth 2 ``` ## Unix Command Wrappers Wrappers around common Unix commands. ### sort\_unix(input, output?) Sort file with `LC_ALL=C sort -u` (in-place if no output specified). ```javascript theme={null} sort_unix("{{Output}}/urls.txt") // In-place sort_unix("{{Output}}/urls.txt", "{{Output}}/urls-sorted.txt") // To new file // Returns: true or false ``` ### wget\_unix(url, output?) Download file with wget. ```javascript theme={null} wget_unix("https://example.com/file.txt", "/tmp/file.txt") // Returns: true or false ``` ### git\_clone(repo, dest?) Clone git repository (shallow clone). ```javascript theme={null} git_clone("https://github.com/user/repo", "/tmp/repo") // Returns: true or false ``` ### zip\_unix(source, dest) Create zip archive using `zip -r`. ```javascript theme={null} zip_unix("{{Output}}", "{{Output}}/archive.zip") // Returns: true or false ``` ### unzip\_unix(source, dest?) Extract zip archive using `unzip`. ```javascript theme={null} unzip_unix("/tmp/archive.zip", "/tmp/extracted") // Returns: true or false ``` ### tar\_unix(source, dest) Create tar.gz archive using `tar -czf`. ```javascript theme={null} tar_unix("{{Output}}", "{{Output}}/archive.tar.gz") // Returns: true or false ``` ### untar\_unix(source, dest?) Extract tar.gz archive using `tar -xzf`. ```javascript theme={null} untar_unix("/tmp/archive.tar.gz", "/tmp/extracted") // Returns: true or false ``` ### diff\_unix(file1, file2, output?) Compare files with diff command. ```javascript theme={null} diff_unix("old.txt", "new.txt") // Returns: diff output diff_unix("old.txt", "new.txt", "changes.diff") // Writes to file, returns: true or false ``` ### sed\_string\_replace(sed\_syntax, source, dest) String replacement with sed `s/old/new/g` syntax. ```javascript theme={null} sed_string_replace("s/http/https/g", "{{Output}}/urls.txt", "{{Output}}/urls-fixed.txt") // Returns: true or false ``` ### wget(url, outputPath) Download file using pure Go (no wget dependency). Supports segmented downloads. ```javascript theme={null} wget("https://example.com/file.zip", "/tmp/file.zip") // Returns: true or false ``` ### git\_clone\_subfolder(git\_url, subfolder, dest) Clone only a specific subfolder from a git repository. ```javascript theme={null} git_clone_subfolder("https://github.com/user/repo.git", "tools/scanner", "/tmp/scanner") // Returns: true or false ``` ### sed\_regex\_replace(sed\_syntax, source, dest) Regex replacement with sed `s/pattern/repl/g` syntax. ```javascript theme={null} sed_regex_replace("s/[0-9]+/NUM/g", "{{Output}}/data.txt", "{{Output}}/data-clean.txt") // Returns: true or false ``` ## Archive Functions (Go) Pure Go implementations for archive operations (no Unix dependencies). ### zip\_dir(source, dest) Zip directory using Go's archive/zip. ```javascript theme={null} zip_dir("{{Output}}", "{{Output}}/archive.zip") // Returns: true or false ``` ### unzip\_dir(source, dest) Unzip archive using Go's archive/zip. ```javascript theme={null} unzip_dir("/tmp/archive.zip", "/tmp/extracted") // Returns: true or false ``` ### extract\_to(source, dest) Auto-detect archive format (.zip, .tar.gz, .tar.bz2, .tar.xz, .tgz) and extract. Removes destination directory first if it exists. ```javascript theme={null} extract_to("{{Output}}/repo.tar.gz", "{{Output}}/repo") // Returns: true or false extract_to("{{Output}}/tools.zip", "{{Output}}/tools") // Returns: true or false ``` ## Diff Functions File comparison and diff extraction. ### extract\_diff(file1, file2) Get lines that are only in file2 (new content). ```javascript theme={null} extract_diff("{{Output}}/old-subs.txt", "{{Output}}/new-subs.txt") // Returns: "newsub1.example.com\nnewsub2.example.com" ``` ## Output Functions Save content and convert data formats. ### save\_content(content, path) Save string content to file. ```javascript theme={null} save_content("Hello World", "{{Output}}/greeting.txt") // Returns: true or false ``` ### jsonl\_to\_csv(source, dest) Convert JSONL file to CSV. ```javascript theme={null} jsonl_to_csv("{{Output}}/assets.jsonl", "{{Output}}/assets.csv") // Returns: true or false ``` ### csv\_to\_jsonl(source, dest) Convert CSV file to JSONL. ```javascript theme={null} csv_to_jsonl("{{Output}}/data.csv", "{{Output}}/data.jsonl") // Returns: true or false ``` ### jsonl\_unique(source, dest, fields) Deduplicate JSONL by hashing selected fields. ```javascript theme={null} jsonl_unique("{{Output}}/httpx.jsonl", "{{Output}}/httpx-unique.jsonl", ["status", "words", "lines"]) // Returns: true or false ``` ### jsonl\_filter(source, dest, fields) Filter JSONL to selected fields only. ```javascript theme={null} jsonl_filter("{{Output}}/httpx.jsonl", "{{Output}}/httpx-filtered.jsonl", "host,status,hash.body_sha256") // Returns: true or false ``` ### jsonl\_rename\_key(source, dest, mappings) Rename keys in JSONL records based on a mapping string. ```javascript theme={null} jsonl_rename_key("{{Output}}/data.jsonl", "{{Output}}/renamed.jsonl", "old_key:new_key,host:domain") // Returns: true or false ``` ## URL Processing Functions URL deduplication and filtering. ### interesting\_urls(src, dest, json\_field?) Deduplicate URLs by hostname+path+params, filter static files and noise patterns. ```javascript theme={null} // Plain text file interesting_urls("{{Output}}/all-urls.txt", "{{Output}}/interesting.txt") // JSONL file with URL in specific field interesting_urls("{{Output}}/katana.jsonl", "{{Output}}/interesting.jsonl", "url") // Returns: true or false ``` ### get\_parent\_url(url) Strip the last path component from a URL. ```javascript theme={null} get_parent_url("https://example.com/api/v1/users") // Returns: "https://example.com/api/v1" ``` ### parse\_url(url, format) Parse a URL and format output using directives (similar to unfurl). ```javascript theme={null} parse_url("https://user:pass@example.com:8080/path?q=1", "%s://%d%p") // Returns: "https://example.com/path" ``` ### parse\_url\_file(input, format, output) Apply `parse_url` formatting to each line of a file and write results. ```javascript theme={null} parse_url_file("{{Output}}/urls.txt", "%d", "{{Output}}/domains.txt") // Returns: true or false ``` ### query\_replace(url, value, mode?) Replace all query parameter values in a URL with a given value. ```javascript theme={null} query_replace("https://example.com/search?q=test&page=1", "FUZZ") // Returns: "https://example.com/search?q=FUZZ&page=FUZZ" ``` ### path\_replace(url, value, position?) Replace a path segment at a given position (or all segments) with a value. ```javascript theme={null} path_replace("https://example.com/api/v1/users", "FUZZ", 2) // Returns: "https://example.com/api/FUZZ/users" ``` ## Markdown Functions Markdown rendering and conversion. ### render\_markdown\_from\_file(path) Render markdown with terminal styling. ```javascript theme={null} render_markdown_from_file("{{Output}}/report.md") // Returns: rendered markdown string ``` ### print\_markdown\_from\_file(path) Print markdown with syntax highlighting to stdout. ```javascript theme={null} print_markdown_from_file("{{Output}}/summary.md") ``` ### convert\_jsonl\_to\_markdown(input\_path, output\_path) Convert JSONL to markdown table and write to file. ```javascript theme={null} convert_jsonl_to_markdown("{{Output}}/assets.jsonl", "{{Output}}/assets.md") // Returns: true or false ``` ### convert\_csv\_to\_markdown(path) Convert CSV to markdown table. ```javascript theme={null} convert_csv_to_markdown("{{Output}}/data.csv") // Returns: markdown table string ``` ### render\_markdown\_report(template\_path, output\_path) Render markdown template with `osm-func` blocks evaluated. ```javascript theme={null} render_markdown_report("{{Templates}}/report.md", "{{Output}}/report.md") // Returns: true or false ``` ### generate\_security\_report(template\_path) Generate security report from template to `{{Output}}/security-report.md` and register as artifact. ```javascript theme={null} generate_security_report("{{MarkdownTemplates}}/security-report-template.md") // Returns: true or false ``` ## Database Functions Database operations for assets, vulnerabilities, and statistics. ### Artifact Registration #### register\_artifact(path, type?) Register file as scan artifact. ```javascript theme={null} register_artifact("{{Output}}/nuclei.json", "nuclei") // Returns: true or false ``` #### store\_artifact(path) Store file as run artifact for current workspace. ```javascript theme={null} store_artifact("{{Output}}/report.md") // Returns: true or false ``` ### Database Updates #### db\_update(table, key, field, value) Update database field. ```javascript theme={null} db_update("workspaces", "{{Workspace}}", "status", "completed") // Returns: true or false ``` ### Asset Import #### db\_import\_asset(workspace, json) Import asset from JSON (upsert - update or insert). ```javascript theme={null} db_import_asset("{{Workspace}}", '{"asset_value":"sub.example.com","asset_type":"subdomain"}') // Returns: true or false ``` #### db\_raw\_insert\_asset(workspace, json) Insert asset from JSON (pure insert, returns ID). ```javascript theme={null} db_raw_insert_asset("{{Workspace}}", '{"asset_value":"api.example.com"}') // Returns: 123 (asset ID) ``` #### db\_import\_asset\_from\_file(workspace, file\_path) Import assets from JSONL file (httpx format supported). ```javascript theme={null} db_import_asset_from_file("{{Workspace}}", "{{Output}}/httpx.jsonl") // Returns: 42 (count) ``` #### db\_quick\_import\_asset(workspace, asset\_value, asset\_type?) Quick import a single asset by value with optional type. ```javascript theme={null} db_quick_import_asset("{{Workspace}}", "sub.example.com", "subdomain") // Returns: true or false ``` #### db\_partial\_import\_asset(workspace, asset\_type, asset\_value) Import a single asset with explicit type. ```javascript theme={null} db_partial_import_asset("{{Workspace}}", "subdomain", "api.example.com") // Returns: true or false ``` #### db\_partial\_import\_asset\_file(workspace, asset\_type, file\_path) Import assets from a plain text file (one per line) with explicit type. ```javascript theme={null} db_partial_import_asset_file("{{Workspace}}", "subdomain", "{{Output}}/subs.txt") // Returns: 150 (count) ``` #### db\_import\_dns\_asset(workspace, file\_path) Import DNS record assets from file. ```javascript theme={null} db_import_dns_asset("{{Workspace}}", "{{Output}}/dns-records.jsonl") // Returns: 50 (count) ``` #### db\_import\_custom\_asset(workspace, file\_path, asset\_type?, source?) Import custom assets from file with optional type and source. ```javascript theme={null} db_import_custom_asset("{{Workspace}}", "{{Output}}/custom.jsonl", "ip", "masscan") // Returns: {imported: 100, skipped: 5} ``` ### Vulnerability Import #### db\_import\_vuln(workspace, json) Import single vulnerability from JSON (nuclei format). ```javascript theme={null} db_import_vuln("{{Workspace}}", '{"template-id":"cve-2024-1234","info":{"name":"CVE","severity":"high"}}') // Returns: true or false ``` #### db\_import\_vuln\_from\_file(workspace, file\_path) Import vulnerabilities from JSONL file (nuclei format). ```javascript theme={null} db_import_vuln_from_file("{{Workspace}}", "{{Output}}/nuclei.jsonl") // Returns: 15 (count) ``` ### Statistics Updates (Write) These functions count lines in a file and update workspace statistics. #### db\_total\_subdomains(path) ```javascript theme={null} db_total_subdomains("{{Output}}/subdomains.txt") // Returns: 150 ``` #### db\_total\_urls(path) ```javascript theme={null} db_total_urls("{{Output}}/urls.txt") // Returns: 5000 ``` #### db\_total\_assets(path) ```javascript theme={null} db_total_assets("{{Output}}/assets.txt") // Returns: 200 ``` #### db\_total\_vulns(path) ```javascript theme={null} db_total_vulns("{{Output}}/vulns.txt") // Returns: 25 ``` #### db\_vuln\_critical(path) ```javascript theme={null} db_vuln_critical("{{Output}}/nuclei.json") // Returns: 2 ``` #### db\_vuln\_high(path) ```javascript theme={null} db_vuln_high("{{Output}}/nuclei.json") // Returns: 5 ``` #### db\_vuln\_medium(path) ```javascript theme={null} db_vuln_medium("{{Output}}/nuclei.json") // Returns: 10 ``` #### db\_vuln\_low(path) ```javascript theme={null} db_vuln_low("{{Output}}/nuclei.json") // Returns: 8 ``` #### db\_total\_ips(path) ```javascript theme={null} db_total_ips("{{Output}}/ips.txt") // Returns: 50 ``` #### db\_total\_links(path) ```javascript theme={null} db_total_links("{{Output}}/links.txt") // Returns: 1000 ``` #### db\_total\_content(path) ```javascript theme={null} db_total_content("{{Output}}/content.txt") // Returns: 500 ``` #### db\_total\_archive(path) ```javascript theme={null} db_total_archive("{{Output}}/archive.txt") // Returns: 100 ``` ### Statistics Queries (Read) These functions read current workspace statistics (no arguments needed). #### db\_select\_total\_subdomains() ```javascript theme={null} db_select_total_subdomains() // Returns: 150 ``` #### db\_select\_total\_urls() ```javascript theme={null} db_select_total_urls() // Returns: 5000 ``` #### db\_select\_total\_assets() ```javascript theme={null} db_select_total_assets() // Returns: 200 ``` #### db\_select\_total\_vulns() ```javascript theme={null} db_select_total_vulns() // Returns: 25 ``` #### db\_select\_vuln\_critical() ```javascript theme={null} db_select_vuln_critical() // Returns: 2 ``` #### db\_select\_vuln\_high() ```javascript theme={null} db_select_vuln_high() // Returns: 5 ``` #### db\_select\_vuln\_medium() ```javascript theme={null} db_select_vuln_medium() // Returns: 10 ``` #### db\_select\_vuln\_low() ```javascript theme={null} db_select_vuln_low() // Returns: 8 ``` ### Data Selection #### db\_select\_assets(workspace, format) Select all assets from workspace. ```javascript theme={null} db_select_assets("{{Workspace}}", "markdown") // Returns: markdown table db_select_assets("{{Workspace}}", "jsonl") // Returns: JSONL string ``` #### db\_select\_assets\_filtered(workspace, status\_code, asset\_type, format) Select assets with filters. ```javascript theme={null} db_select_assets_filtered("{{Workspace}}", "200", "subdomain", "jsonl") // Returns: filtered JSONL ``` #### db\_select\_vulnerabilities(workspace, format) Select all vulnerabilities from workspace. ```javascript theme={null} db_select_vulnerabilities("{{Workspace}}", "markdown") // Returns: markdown table ``` #### db\_select\_vulnerabilities\_filtered(workspace, severity, asset\_value, format) Select vulnerabilities with filters. ```javascript theme={null} db_select_vulnerabilities_filtered("{{Workspace}}", "critical", "", "jsonl") // Returns: filtered JSONL ``` #### db\_select(sql\_query, format) Execute SELECT query with format. ```javascript theme={null} db_select("SELECT * FROM assets WHERE workspace = '{{Workspace}}' LIMIT 10", "markdown") ``` #### db\_select\_to\_file(sql\_query, dest) Execute SELECT and write markdown to file. ```javascript theme={null} db_select_to_file("SELECT * FROM assets", "{{Output}}/assets.md") // Returns: true or false ``` #### db\_select\_to\_jsonl(sql\_query, fields, dest) Execute SELECT and write JSONL with specified fields. ```javascript theme={null} db_select_to_jsonl("SELECT * FROM assets", "asset_value,status_code", "{{Output}}/assets.jsonl") // Returns: true or false ``` ### Diff Tracking #### db\_asset\_diff(workspace) Get asset diff as JSONL string (new/changed assets since last scan). ```javascript theme={null} db_asset_diff("{{Workspace}}") // Returns: JSONL string ``` #### db\_vuln\_diff(workspace) Get vulnerability diff as JSONL string. ```javascript theme={null} db_vuln_diff("{{Workspace}}") // Returns: JSONL string ``` #### db\_asset\_diff\_to\_file(workspace, dest) Write asset diff to JSONL file. ```javascript theme={null} db_asset_diff_to_file("{{Workspace}}", "{{Output}}/asset-diff.jsonl") // Returns: true or false ``` #### db\_vuln\_diff\_to\_file(workspace, dest) Write vulnerability diff to JSONL file. ```javascript theme={null} db_vuln_diff_to_file("{{Workspace}}", "{{Output}}/vuln-diff.jsonl") // Returns: true or false ``` ### Run Status #### run\_status(workspace, format) Query run records for a workspace. ```javascript theme={null} run_status("{{Workspace}}", "markdown") // Returns: markdown table of runs run_status("{{Workspace}}", "jsonl") // Returns: JSONL string ``` #### run\_status\_by\_uuid(uuid, format) Query a specific run by UUID. ```javascript theme={null} run_status_by_uuid("abc123-uuid", "markdown") // Returns: markdown table ``` ### Event Log Management #### db\_reset\_event\_logs(workspace?, topic\_pattern?) Reset (delete) event logs, optionally filtered by workspace and topic pattern. ```javascript theme={null} db_reset_event_logs() // Reset all event logs db_reset_event_logs("{{Workspace}}") // Reset for specific workspace db_reset_event_logs("{{Workspace}}", "discovery.*") // Reset matching topic pattern // Returns: {reset: 42, total: 100} ``` ### Runtime Export #### runtime\_export() Export scan and workspace state to `run-state.json`. ```javascript theme={null} runtime_export() // Returns: true or false ``` ## Installer Functions Download and install packages. ### go\_getter(url, dest) Download files/repos using go-getter (supports git, http, s3, gcs, etc.). ```javascript theme={null} go_getter("https://github.com/user/repo.git?ref=main", "{{Output}}/repo") go_getter("s3::https://bucket.s3.amazonaws.com/file.zip", "/tmp/file.zip") // Returns: true or false ``` ### go\_getter\_with\_sshkey(ssh\_key\_path, git\_url, dest) Clone git repo via SSH with specified key. ```javascript theme={null} go_getter_with_sshkey("~/.ssh/id_rsa", "git@github.com:user/private-repo.git", "{{Output}}/repo") // Returns: true or false ``` ### nix\_install(package, dest?) Install package via Nix package manager. ```javascript theme={null} nix_install("nuclei", "{{Binaries}}") // Returns: true or false ``` ### filepath\_installer(local\_path, tool\_name, dest?) Install a binary from a local file path by copying it to the destination. ```javascript theme={null} filepath_installer("/tmp/downloads/nuclei", "nuclei", "{{Binaries}}") // Returns: true or false ``` ## Environment Functions Environment variable operations. ### os\_getenv(name) Get environment variable. ```javascript theme={null} os_getenv("HOME") // Returns: "/home/user" os_getenv("API_KEY") // Returns: "secret" or "" ``` ### os\_setenv(name, value) Set environment variable. ```javascript theme={null} os_setenv("API_KEY", "new-secret") // Returns: true or false ``` ## SARIF Functions Parse and import SARIF (Static Analysis Results Interchange Format) output from SAST tools. ### db\_import\_sarif(workspace, file\_path) Import vulnerabilities from a SARIF file into the database. Supports Semgrep, Trivy, Kingfisher, Bearer, and other SARIF-producing tools. ```javascript theme={null} db_import_sarif("{{Workspace}}", "{{Output}}/semgrep.sarif") // Returns: {imported: 15, skipped: 2, errors: 0} ``` ### convert\_sarif\_to\_markdown(input\_path, output\_path) Convert SARIF file to readable markdown tables. ```javascript theme={null} convert_sarif_to_markdown("{{Output}}/scan.sarif", "{{Output}}/findings.md") // Returns: true or false ``` ## Nmap/Port Functions Port scanning and result processing. ### nmap\_to\_jsonl(input\_path, output\_path) Convert nmap XML or gnmap output to JSONL format. ```javascript theme={null} nmap_to_jsonl("{{Output}}/nmap.xml", "{{Output}}/ports.jsonl") // Returns: true or false ``` ### run\_nmap(target, flags?, output?) Execute nmap scan and auto-convert results to JSONL. ```javascript theme={null} run_nmap("192.168.1.0/24") // Returns: path to output JSONL run_nmap("10.0.0.1", "-sV -p 80,443", "{{Output}}/nmap") // Returns: path to output JSONL ``` ### db\_import\_port\_assets(workspace, file\_path, source?) Import port scan JSONL into database as IP assets. ```javascript theme={null} db_import_port_assets("{{Workspace}}", "{{Output}}/ports.jsonl", "nmap") // Returns: {imported: 50, skipped: 0, errors: 0} ``` ## Tmux Functions Manage background processes via tmux sessions. ### tmux\_run(command, session\_name?) Create a detached tmux session running a command. Auto-generates a `bosm-` name if not provided. ```javascript theme={null} tmux_run("nuclei -l targets.txt -o results.txt") // Returns: "bosm-a1b2c3d4" (session name) tmux_run("long-scan.sh", "my-scan") // Returns: "my-scan" ``` ### tmux\_capture(session\_name) Capture the current pane output from a tmux session. Pass `"all"` to capture all sessions. ```javascript theme={null} tmux_capture("my-scan") // Returns: current output as string tmux_capture("all") // Returns: output from all sessions ``` ### tmux\_send(session\_name, command) Send keystrokes to a running tmux session. ```javascript theme={null} tmux_send("my-scan", "q") // Returns: true or false tmux_send("my-scan", "C-c") // Send Ctrl+C ``` ### tmux\_kill(session\_name) Kill a tmux session. ```javascript theme={null} tmux_kill("my-scan") // Returns: true or false ``` ### tmux\_list() List all tmux session names. ```javascript theme={null} tmux_list() // Returns: ["bosm-a1b2c3d4", "my-scan", ...] ``` ## SSH/Sync Functions Remote execution and file synchronization. ### ssh\_exec(host, command, user?, key\_path?, password?, port?) Execute a command on a remote host via SSH (uses connection pooling). ```javascript theme={null} ssh_exec("10.0.0.1", "whoami") // Returns: "root" ssh_exec("10.0.0.1", "ls /opt", "admin", "~/.ssh/id_rsa", "", 22) // Returns: command output ``` ### ssh\_rsync(host, src, dest, user?, key\_path?, password?, port?) Copy files to/from a remote host via rsync over SSH. ```javascript theme={null} ssh_rsync("10.0.0.1", "{{Output}}/results/", "/tmp/results/", "admin", "~/.ssh/id_rsa") // Returns: true or false ``` ### sync\_from\_master(src, dest) Pull files from the master node. Falls back to local copy if not in distributed mode. ```javascript theme={null} sync_from_master("{{BaseFolder}}/wordlists/", "{{Output}}/wordlists/") // Returns: true or false ``` ### sync\_from\_worker(identifier, ip, src, dest) Sync files from a specific worker node. ```javascript theme={null} sync_from_worker("wosm-abc123", "10.0.0.2", "/tmp/results/", "{{Output}}/worker-results/") // Returns: true or false ``` ### rsync\_to\_worker(identifier, ip, src, dest) Push files to a specific worker node. ```javascript theme={null} rsync_to_worker("wosm-abc123", "10.0.0.2", "{{Output}}/config/", "/tmp/config/") // Returns: true or false ``` ## Script Execution Functions Execute Python and TypeScript code from workflows. ### exec\_python(code) Run inline Python code. Prefers `uv` → `python3` → `python`. ```javascript theme={null} exec_python('import json; print(json.dumps({"status": "ok"}))') // Returns: '{"status": "ok"}' ``` ### exec\_python\_file(path) Run a Python file. Prefers `uv` → `python3` → `python`. ```javascript theme={null} exec_python_file("{{Output}}/scripts/analyze.py") // Returns: script stdout ``` ### exec\_ts(code) Run inline TypeScript code via `bun -e`. ```javascript theme={null} exec_ts('console.log("Hello from TypeScript")') // Returns: "Hello from TypeScript" ``` ### exec\_ts\_file(path) Run a TypeScript file via `bun run`. ```javascript theme={null} exec_ts_file("{{Output}}/scripts/process.ts") // Returns: script stdout ``` ## LLM Functions Invoke LLM models from workflows. ### llm\_invoke(message) Send a simple message to the configured LLM and return the response. ```javascript theme={null} llm_invoke("Summarize these findings: " + read_file("{{Output}}/results.txt")) // Returns: LLM response as string ``` ### llm\_invoke\_custom(message, body\_json) Send a message with a custom POST body template. Use `{{message}}` as placeholder in the body. ```javascript theme={null} llm_invoke_custom("Analyze this", '{"model": "gpt-4", "messages": [{"role": "user", "content": "{{message}}"}]}') // Returns: LLM response as string ``` ### llm\_conversations(msg1, msg2, ...) Multi-turn LLM conversation. Messages use `"role:content"` format. ```javascript theme={null} llm_conversations("system:You are a security analyst", "user:What is XSS?") // Returns: LLM response as string ``` ## Agent/Distributed Functions Run ACP agents and distribute execution across nodes. ### run\_agent(message, agent\_name?) Run an ACP agent and return its output. Defaults to `claude-code`. ```javascript theme={null} run_agent("Analyze the scan results in {{Output}}") // Returns: agent output run_agent("Review this code", "codex") // Returns: agent output ``` ### run\_on\_master(action, ...args) Execute a function or command on the master node via Redis. ```javascript theme={null} run_on_master("func", 'db_import_sarif("ws", "/path/to/file.sarif")') // Returns: true or false ``` ### run\_on\_worker(scope, action, ...args) Execute a function or command on worker nodes. ```javascript theme={null} run_on_worker("all", "func", 'log_info("hello from workers")') // Returns: true or false ``` ## Module Control Functions Control module execution flow. ### skip(message?) Abort remaining steps in the current module. The flow continues to the next module. ```javascript theme={null} skip("No targets found, skipping module") // Raises SkipModuleError skip() // Skip without message ``` ### run\_module(module, target, params?) Run an osmedeus module programmatically. ```javascript theme={null} run_module("subdomain", "example.com") // Returns: execution output run_module("portscan", "example.com", "threads=20") // With custom params ``` ### run\_flow(flow, target, params?) Run an osmedeus flow programmatically. ```javascript theme={null} run_flow("general", "example.com") // Returns: execution output run_flow("recon", "example.com", "threads=10") // With custom params ``` ## Snapshot Functions Workspace export and import as compressed ZIP archives. ### snapshot\_export(workspace, dest?) Export a workspace as a ZIP archive. Returns the path to the created file. ```javascript theme={null} snapshot_export("{{Workspace}}") // Returns: "/path/to/snapshot.zip" snapshot_export("{{Workspace}}", "/tmp/backup.zip") // Returns: "/tmp/backup.zip" ``` ### snapshot\_import(source) Import a workspace from a ZIP file or URL. Returns the workspace name. ```javascript theme={null} snapshot_import("/tmp/snapshot.zip") // Returns: "example.com" snapshot_import("https://example.com/snapshot.zip") // Returns: "example.com" ``` ## Authentication Functions System authentication operations. ### sudo\_auth(password?, keepalive?) Authenticate sudo credentials and optionally keep them alive for the session. ```javascript theme={null} sudo_auth() // Prompt for password sudo_auth("mypassword", true) // Authenticate and keep credentials alive // Returns: true or false ``` ## Usage Examples ### Conditional Execution ```yaml theme={null} - name: scan-if-hosts type: bash pre_condition: 'file_length("{{Output}}/hosts.txt") > 0 && file_exists("{{Output}}/hosts.txt")' command: nuclei -l {{Output}}/hosts.txt ``` ### Chain Functions ```yaml theme={null} - name: process type: function functions: - log_info("Starting processing") - save_content("processing", "{{Output}}/status.txt") - log_info("Processing complete") ``` ### Export Results ```yaml theme={null} - name: count type: function function: file_length("{{Output}}/results.txt") exports: result_count: "{{result}}" - name: report type: function function: log_info("Found " + result_count + " results") ``` ### Decision Based on Function ```yaml theme={null} - name: check-size type: function function: file_length("{{Output}}/data.txt") exports: size: "{{result}}" decision: switch: "{{size}}" cases: "0": { goto: no-data } default: { goto: process-data } ``` ### HTTP API Integration ```yaml theme={null} - name: submit-results type: function function: | http_post("https://api.example.com/results", '{"target": "{{Target}}", "count": ' + result_count + '}') ``` ### Event Generation ```yaml theme={null} - name: emit-discoveries type: function functions: - generate_event_from_file("{{Workspace}}", "discovery", "amass", "subdomain", "{{Output}}/subdomains.txt") - log_info("Emitted subdomain discovery events") ``` ### Database Reporting ```yaml theme={null} - name: generate-report type: function functions: - save_content("# Vulnerability Report\n\n", "{{Output}}/report.md") - append_file("{{Output}}/report.md", db_select_vulnerabilities("{{Workspace}}", "markdown")) - log_info("Report generated at {{Output}}/report.md") ``` ## CLI Testing ```bash theme={null} # Test file functions osmedeus func e 'file_exists("/etc/passwd")' osmedeus func e 'file_length("/etc/hosts")' # Test string functions osmedeus func e 'trim(" hello ")' osmedeus func e 'split("a,b,c", ",")' # Test with target osmedeus func e 'log_info("Target: " + target)' -t example.com # Test JSON osmedeus func e 'jq("{\"a\":1}", ".a")' # Test event generation osmedeus func e 'generate_event("test-workspace", "test.topic", "cli", "test", "data")' # Bulk processing osmedeus func e 'log_info("Processing: " + target)' -T targets.txt -c 10 # List functions osmedeus func list -s "file" osmedeus func list -s "event" --example ``` ## Next Steps * [Functions Overview](overview) - How to use functions in workflows * [Step Types](/workflows/step-types) - Function steps * [Control Flow](/workflows/control-flow) - Conditions and decisions # Basic Configuration Source: https://docs.osmedeus.org/getting-started/basic-setup Osmedeus configuration management and settings Osmedeus configuration is managed through YAML file which store in `~/osmedeus-base/osm-settings.yaml` by default. You can modify the configuration file directly or use the Configuration CLI. ## 1. Edit the YAML file directly Descriptive alt text See the full YAML config with comments below to understand what each field means. For increase the quality of the scan, you mostly need to modify the field `global_vars.[keyname]` which will hold the global variables that can be used in the workflows. Some example keys can be `GITHUB_API_KEY, SHODAN_API_KEY, PASSIVETOTAL_API_KEY, etc.` ```yaml theme={null} # Osmedeus Configuration File # This file contains all available configuration options for osmedeus. # Copy this file to ~/osmedeus-base/osm-settings.yaml and customize as needed. # ============================================================================= # Base Folder # ============================================================================= # Root directory for all osmedeus data (workflows, binaries, data, etc.) # Environment variables like $HOME are automatically expanded base_folder: $HOME/osmedeus-base # ============================================================================= # Environment Paths # ============================================================================= # Directory paths for various osmedeus components # Use {{base_folder}} to reference the base_folder value above environments: # Path to binary executables (tools like nmap, ffuf, etc.) external_binaries_path: "{{base_folder}}/external-binaries" # Data directory for storing assets, wordlists, etc. external_data: "{{base_folder}}/external-data" # External configuration files (nuclei templates, etc.) external_configs: "{{base_folder}}/external-configs" # Output directory for scan workspaces # Each target gets its own subdirectory here workspaces: $HOME/workspaces-osmedeus # Directory containing workflow YAML files # Subdirectories: flows/, modules/ workflows: "{{base_folder}}/workflows" # Directory for workspace snapshots (zip archives) # Used by the snapshot-download API endpoint snapshot: "{{base_folder}}/snapshot" # Directory for markdown report templates # Used by render_markdown_report() function markdown_report_templates: "{{base_folder}}/markdown-report-templates" # Directory for external agent configurations # Used for LLM Agent commands, skills, and related configurations external_agent_configs: "{{base_folder}}/external-agent-configs" # ============================================================================= # Database Configuration # ============================================================================= # Osmedeus supports SQLite (default) and PostgreSQL database: # Database engine: "sqlite" or "postgresql" db_engine: sqlite # SQLite: Path to the database file # Ignored when using PostgreSQL db_path: "{{base_folder}}/database-osm.sqlite" # PostgreSQL connection settings # Only used when db_engine is "postgresql" host: localhost port: 5432 username: osmedeus password: osmedeus db_name: osmedeus # Connection timeout in seconds connection_timeout: 60 # PostgreSQL SSL mode: disable, require, verify-ca, verify-full ssl_mode: disable # ============================================================================= # Server Configuration # ============================================================================= # REST API server settings for the web interface server: # Host to bind the server to # Use "0.0.0.0" to listen on all interfaces # Use "127.0.0.1" to listen only on localhost host: "0.0.0.0" # Port number for the API server port: 8002 # Path to serve static UI files # Default: {{base_folder}}/ui/ - if this directory exists, it will be served at /ui # Set to empty string to disable UI serving ui_path: "{{base_folder}}/ui/" # Random prefix for workspace static files (auto-generated 16 chars if empty) # Used as URL path segment for direct access to workspaces folder workspace_prefix_key: "" # Authentication credentials (map of username:password) # Supports multiple users simple_user_map_key: osmedeus: osmedeus-admin # JWT (JSON Web Token) settings jwt: # Secret key for signing JWT tokens # IMPORTANT: Use a strong, unique secret in production! secret_signing_key: change-this-secret-in-production # Token expiration time in minutes expiration_minutes: 180 # License type shown in HTTP Server header and /server-info endpoint license: "open-source" # ============================================================================= # Scan Tactic Configuration # ============================================================================= # Thread counts for different scan intensity levels # Higher values = faster but more aggressive scans # Lower values = slower but gentler on target systems scan_tactic: # Aggressive/fast mode - maximum parallelism # Used with: osmedeus scan -t target --tactic aggressive aggressive: 40 # Default/normal mode - balanced approach # Used when no tactic is specified default: 10 # Gentle/thorough mode - minimal parallelism # Used with: osmedeus scan -t target --tactic gently gently: 5 # ============================================================================= # Redis Configuration (Optional) # ============================================================================= # Redis is required for distributed scanning mode # Leave host empty to disable Redis redis: # Redis server hostname # Leave empty to disable distributed mode host: "" # Redis server port port: 6379 # Redis authentication (if required) username: "" password: "" # Redis database number (0-15) db: 0 # Connection timeout in seconds connection_timeout: 60 # ============================================================================= # Global Variables # ============================================================================= # User-defined variables available in workflows via {{VARIABLE_NAME}} # Variables can optionally be exported to environment variables # Use _API_KEY suffix for secrets to indicate sensitive values # # Format: # VARIABLE_NAME: # value: "the-value" # as_env: true # Optional: export as env var (default: true) # # Example usage in workflows: # - bash: "echo {{GITHUB_API_KEY}}" # - bash: "shodan search $SHODAN_API_KEY" # Uses env var global_vars: # GitHub personal access token for API access GITHUB_API_KEY: value: "" as_env: true # Exports as GITHUB_API_KEY # Shodan API key for passive reconnaissance SHODAN_API_KEY: value: "" as_env: true # Exports as SHODAN_API_KEY # Censys API key for certificate/host search CENSYS_API_KEY: value: "" as_env: true # Exports as CENSYS_API_KEY # PassiveTotal API key for passive DNS/WHOIS PASSIVETOTAL_API_KEY: value: "" as_env: true # Exports as PASSIVETOTAL_API_KEY # Add more API keys as needed (use _API_KEY suffix for secrets) # ============================================================================= # Notification Configuration # ============================================================================= # Send notifications when scans complete or find interesting results notification: # Notification provider: "telegram" (future: slack, discord, webhook) provider: telegram # Master switch to enable/disable all notifications enabled: false # Telegram bot settings # Create a bot via @BotFather and get the token # Get your chat ID by messaging @userinfobot telegram: # Bot token from @BotFather bot_token: "" # Chat ID to send messages to (can be user or group) chat_id: 0 # Enable Telegram notifications enabled: false # ============================================================================= # Cloud Storage Configuration (Optional) # ============================================================================= # S3-compatible storage for backing up scan results # Supports AWS S3, MinIO, Google Cloud Storage, DigitalOcean Spaces, etc. storage: # Storage provider: "s3", "minio", "gcs", "spaces", etc. provider: s3 # Storage endpoint URL # AWS S3: Leave empty or use region-specific endpoint # MinIO: "http://localhost:9000" # DigitalOcean: "https://nyc3.digitaloceanspaces.com" endpoint: "" # Access credentials access_key_id: "" secret_access_key: "" # Bucket name for storing results bucket: "" # Cloud region (e.g., us-east-1, eu-west-1) region: us-east-1 # Use SSL/TLS for connections use_ssl: true # Enable cloud storage uploads enabled: false # ============================================================================= # LLM Configuration (Optional) # ============================================================================= # Large Language Model settings for AI-powered features # Supports providers like Ollama, OpenAI, Anthropic, etc. # Multiple providers can be configured for automatic rotation on error/rate limit llm_config: # List of LLM providers (rotates to next on error/rate limit) llm_providers: # Primary provider (used first) - provider: ollama base_url: "http://localhost:11434/v1/chat/completions" auth_token: "" model: "gpt-oss:120b-cloud" # Backup provider example (uncomment to enable rotation) # - provider: openai # base_url: "https://api.openai.com/v1/chat/completions" # auth_token: "sk-your-api-key" # model: "gpt-4" # Enable LLM tool call features enabled_tool_call: false # Maximum number of tokens to generate max_tokens: 1000 # Temperature for sampling temperature: 0.7 # Top-k sampling top_k: 50 # Top-p sampling top_p: 0.9 # Number of completions to generate n: 1 # Maximum number of retries for failed requests max_retries: 3 # Timeout for API requests timeout: 120s # Enable streaming responses stream: false # Enable structured JSON output format structured_json_format: false # System prompt for the LLM system_prompt: "" # Custom headers for API requests custom_headers: "" ``` ## 2. Use Configuration CLI Osmedeus provides a CLI tool to manage the configuration file via `osmedeus config` command. Descriptive alt text ```bash theme={null} base_folder = $HOME/osmedeus-base database.connection_timeout = 60 database.db_engine = sqlite database.db_name = osmedeus database.db_path = {{base_folder}}/database-osm.sqlite database.host = localhost database.password = [REDACTED] database.port = 5432 database.ssl_mode = disable database.username = osmedeus environments.external_agent_configs = {{base_folder}}/external-agent-configs environments.external_binaries_path = {{base_folder}}/external-binaries environments.external_configs = {{base_folder}}/external-configs environments.external_data = {{base_folder}}/external-data environments.markdown_report_templates = {{base_folder}}/markdown-report-templates environments.snapshot = {{base_folder}}/snapshot environments.workflows = {{base_folder}}/workflows environments.workspaces = $HOME/workspaces-osmedeus global_vars.CENSYS_API_KEY.as_env = [REDACTED] global_vars.CENSYS_API_KEY.value = [REDACTED] global_vars.GITHUB_API_KEY.as_env = [REDACTED] global_vars.GITHUB_API_KEY.value = [REDACTED] global_vars.PASSIVETOTAL_API_KEY.as_env = [REDACTED] global_vars.PASSIVETOTAL_API_KEY.value = [REDACTED] global_vars.SHODAN_API_KEY.as_env = [REDACTED] global_vars.SHODAN_API_KEY.value = [REDACTED] llm_config.custom_headers = llm_config.enabled_tool_call = false llm_config.llm_providers.0.auth_token = [REDACTED] llm_config.llm_providers.0.base_url = http://localhost:11434/v1/chat/completions llm_config.llm_providers.0.model = gpt-oss:120b-cloud llm_config.llm_providers.0.provider = ollama llm_config.max_retries = 3 llm_config.max_tokens = [REDACTED] llm_config.n = 1 llm_config.stream = false llm_config.structured_json_format = false llm_config.system_prompt = llm_config.temperature = 0.7 llm_config.timeout = 120s llm_config.top_k = 50 llm_config.top_p = 0.9 notification.enabled = false notification.provider = telegram notification.telegram.bot_token = [REDACTED] notification.telegram.chat_id = 0 notification.telegram.enabled = false redis.connection_timeout = 60 redis.db = 0 redis.host = redis.password = [REDACTED] redis.port = 6379 redis.username = scan_tactic.aggressive = 40 scan_tactic.default = 10 scan_tactic.gently = 5 server.host = 0.0.0.0 server.jwt.expiration_minutes = 180 server.jwt.secret_signing_key = [REDACTED] server.license = open-source server.password = [REDACTED] server.port = 8002 server.simple_user_map_key.osmedeus = [REDACTED] server.ui_path = {{base_folder}}/ui/ server.username = osmedeus server.workspace_prefix_key = [REDACTED] storage.access_key_id = [REDACTED] storage.bucket = storage.enabled = false storage.endpoint = storage.provider = s3 storage.region = us-east-1 storage.secret_access_key = [REDACTED] storage.use_ssl = true ``` # CLI Interface Source: https://docs.osmedeus.org/getting-started/cli Command-line interface for Osmedeus Descriptive alt text This document describes the command-line interface (CLI) for Osmedeus which is the main entry point for running workflows and modules and some other utilities. It provides usage examples and explanations for each command. Below are all available commands: ```bash theme={null} j3ssie ▶ osmedeus -h Usage: osmedeus [flags] osmedeus [command] Available Commands: agent Run an ACP agent interactively assets Query and list discovered assets client Interact with a remote osmedeus server cloud Cloud infrastructure management commands completion Generate the autocompletion script for the specified shell config Manage osmedeus configuration db Database management commands eval Evaluate a script (shorthand for 'func eval') function Execute and test utility functions health Check and fix environment health (alias for 'osmedeus install validate') help Help about any command install Install workflows, base folder, or binaries run Execute a workflow scan Execute a workflow (alias for 'run') serve Start the Osmedeus web server snapshot Export and import workspace snapshots uninstall Remove Osmedeus installation (base folder, workspaces, and binary) update Update osmedeus to the latest version version Print version information worker Worker node commands for distributed scanning workflow Manage workflows ``` ## 1. Osmedeus Run (aliases: scan) Execute a workflow against one or more targets Descriptive alt text Descriptive alt text Descriptive alt text ```bash theme={null} ▷ Examples # Run against a single target osmedeus run -f recon-workflow -t example.com # Run against multiple targets osmedeus run -m simple-module -t target1.com -t target2.com # Run with stdin input with concurrency cat list-of-urls.txt | osmedeus run -m simple-module --concurrency 10 # Combine multiple input methods echo "extra.com" | osmedeus run -m simple-module -t main.com -T more-targets.txt # Run with custom parameters osmedeus run -m simple-module -t example.com --params 'threads=20' # Run with custom base folder osmedeus run --base-folder /opt/osmedeus-base -f recon-workflow -t example.com # Run with timeout (cancel if exceeds) osmedeus run -m recon -t example.com --timeout 2h # Repeat run every hour continuously osmedeus run -m recon -t example.com --repeat --repeat-wait-time 1h # Run multiple modules in sequence osmedeus run -m subdomain -m portscan -m vulnscan -t example.com # Combine timeout with repeat mode osmedeus run -m recon -t example.com --timeout 3h --repeat --repeat-wait-time 30m # Dry-run mode (preview without executing) osmedeus run -m recon -t example.com --dry-run # Run module from stdin (pipe YAML) cat module.yaml | osmedeus run --std-module -t example.com # Load parameters from YAML/JSON file osmedeus run -m recon -t example.com --params-file params.yaml # Custom workspace path osmedeus run -m recon -t example.com --workspace /custom/workspace # Skip heuristics checks osmedeus run -m recon -t example.com --heuristics-check none # Concurrent targets from file osmedeus run -m recon -T targets.txt --concurrency 5 ▷ Module from URL # Run module from URL osmedeus run --module-url https://example.com/module.yaml -t example.com # Run module from GitHub (public) osmedeus run --module-url https://raw.githubusercontent.com/user/repo/main/module.yaml -t example.com # Run module from private GitHub repo (requires GH_TOKEN or GITHUB_API_KEY) osmedeus run --module-url https://github.com/user/private-repo/blob/main/module.yaml -t example.com ▷ Chunk Mode (split large target files across machines) # View chunk info for target file osmedeus run -m recon -T targets.txt --chunk-size 100 # Run specific chunk (0-indexed) osmedeus run -m recon -T targets.txt --chunk-size 100 --chunk-part 2 # Split into 4 equal chunks and run chunk 0 osmedeus run -m recon -T targets.txt --chunk-count 4 --chunk-part 0 # Distributed processing across machines osmedeus run -m recon -T targets.txt --chunk-size 250 --chunk-part 0 # Machine 1 osmedeus run -m recon -T targets.txt --chunk-size 250 --chunk-part 1 # Machine 2 ▷ Queue Mode (defer execution for later processing) # Queue a run for later processing osmedeus run --queue -m recon -t example.com # Queue with file target osmedeus run --queue -m recon -T targets.txt # Process queued tasks (alias for 'osmedeus worker queue run') osmedeus run --queue-run # Process queued tasks with concurrency osmedeus run --queue-run --concurrency 3 ▷ Module Exclusion # Exclude specific module(s) from a flow (exact match, repeatable) osmedeus run -f general -t example.com -x subdomain-enum # Exclude multiple modules osmedeus run -f general -t example.com -x subdomain-enum -x portscan # Fuzzy-exclude modules whose name contains a substring osmedeus run -f general -t example.com -X vuln # Combine exact and fuzzy exclusion osmedeus run -f general -t example.com -x portscan -X brute ▷ Webhook Mode (register a trigger instead of running immediately) # Register a webhook trigger for a module osmedeus run --as-webhook -m recon -t example.com # Register with an authentication key osmedeus run --as-webhook -m recon -t example.com --webhook-auth-key my-secret-key # Register a flow webhook osmedeus run --as-webhook -f general -t example.com --webhook-auth-key my-secret-key ▷ Cron Schedule Mode (create a recurring schedule instead of running immediately) # Create a cron schedule (daily at 2 AM) osmedeus run --as-cron '0 2 * * *' -m recon -t example.com # Schedule a flow to run every 6 hours osmedeus run --as-cron '0 */6 * * *' -f general -t example.com # Schedule for multiple targets (weekly on Monday) osmedeus run --as-cron '0 0 * * 1' -m recon -T targets.txt ``` ## 2. Osmedeus Function Eval Execute and test utility functions available in workflows Descriptive alt text Descriptive alt text ```bash theme={null} ▶ Subcommands • list - List all available functions • eval (e) - Evaluate scripts with template rendering ▷ Examples # List all available functions osmedeus func list # Evaluate a simple function osmedeus func eval 'trim(" hello ")' # Short alias for eval osmedeus func e 'log_info("Hello World")' # Use with target variable osmedeus func e 'fileExists("{{target}}")' -t /tmp/test.txt # Print markdown file with syntax highlighting osmedeus func e 'print_markdown_from_file("README.md")' # Multi-line script with variable osmedeus func e 'var x = trim(" test "); log_info(x); x' # Make HTTP request osmedeus func e 'httpRequest("https://api.example.com", "GET", {}, "")' # With custom params osmedeus func e 'log_info("{{host}}:{{port}}")' --params 'host=localhost' --params 'port=8080' # Use -f flag for shell path autocompletion on file arguments osmedeus func e -f trim " hello world " osmedeus func e -f fileExists /tmp osmedeus func e -t example.com -f log_info "Processing {{target}}" # Query database with SQL osmedeus func e 'db_select("SELECT severity, COUNT(*) FROM vulnerabilities GROUP BY severity", "markdown")' # Query filtered assets from database osmedeus func e 'db_select_assets_filtered("example.com", 200, "subdomain", "jsonl")' # Read script from stdin echo 'log_info("hello")' | osmedeus func e --stdin # Alternative stdin syntax echo 'trim(" test ")' | osmedeus func e - ▷ Bulk Processing # Process multiple targets from file osmedeus func e 'log_info("Processing: " + target)' -T targets.txt # Function from file with targets osmedeus func e --function-file check.js -T targets.txt # With concurrency osmedeus func e 'httpGet("https://" + target)' -T targets.txt -c 10 # Combined with params osmedeus func e 'log_info(prefix + target)' -T targets.txt --params 'prefix=test_' -c 5 ``` ## 3. Installation Command One-liner to install workflows, base folder, or binaries from various sources. ```bash theme={null} ◆ Description Install workflows, base folder, or binaries from various sources. ▶ Subcommands • workflow - Install workflows from git URL, zip URL, or local zip • base - Install base folder (backs up and restores database) • binary - Install binaries from registry • env - Add binaries path to shell configuration • validate - Check and fix environment health ▷ Examples # List available binaries (direct-fetch mode) osmedeus install binary --list-registry-direct-fetch # List available binaries (nix-build mode) osmedeus install binary --list-registry-nix-build # Install specific binaries osmedeus install binary --name nuclei --name httpx # Install all required binaries osmedeus install binary --all # Install all binaries including optional ones osmedeus install binary --all --install-optional # Check if binaries are installed osmedeus install binary --all --check # Install Nix package manager osmedeus install binary --nix-installation # Install binary via Nix osmedeus install binary --name nuclei --nix-build-install # Install all binaries via Nix osmedeus install binary --all --nix-build-install # Install workflows from git, zip URL, or local zip file osmedeus install workflow https://github.com/user/osmedeus-workflows.git osmedeus install workflow http:///workflow-osmedeus.zip osmedeus install workflow local-file-workflow-osmedeus.zip # Install base folder from git, zip URL, or local zip file osmedeus install base https://github.com/user/osmedeus-base.git osmedeus install base http:///osmedeus-base.zip osmedeus install base local-file-osmedeus-base.zip ``` ## 4. Database Management View and manage the database, including clean up, and schema operations. cli-db-list cli-db-view ```bash theme={null} ◆ Description List all database tables with their row counts, or list records from a specific table with pagination support. ▶ Subcommands • list (ls) - List database tables and row counts (default) • seed - Seed database with sample data • clean - Remove all data from database • migrate - Run database migrations • index - Index resources from filesystem to database ▶ Options (list) -t, --table Table name to list records from --offset Number of records to skip (default: 0) --limit Maximum records to return (default: 50) --list-columns List all available columns for the specified table --exclude-columns Comma-separated column names to exclude from output --columns Comma-separated columns to display --all Show all columns including hidden ones --search Search all columns for substring --where Filter records (key=value format, repeatable) --refresh Auto-refresh interval (e.g., 5s, 1m) --clear Clear all records from specified table (requires --force) ▶ Valid Tables runs, step_results, artifacts, assets, event_logs, schedules, workspaces, vulnerabilities ▷ List Examples # List all tables with row counts osmedeus db list # List records from runs table osmedeus db list -t runs # List available columns for assets table osmedeus db list -t assets --list-columns # List assets excluding specific columns osmedeus db list -t assets --exclude-columns id,created_at,updated_at # List assets with pagination osmedeus db list -t assets --offset 0 --limit 10 # Get next page of results osmedeus db list -t assets --offset 10 --limit 10 # Auto-refresh every 5 seconds osmedeus db list -t runs --refresh 5s # JSON output osmedeus db list -t runs --json ▷ Seed Examples # Seed database with sample data osmedeus db seed ▷ Clean Examples # Remove all data from database (requires --force) osmedeus db clean --force # Remove all data including workspace files osmedeus db clean --force --clean-ws ▷ Migrate Examples # Run database migrations osmedeus db migrate ▷ Index Examples # Index workflows from filesystem to database osmedeus db index workflow # Force re-index all workflows osmedeus db index workflow --force ``` ## 5. Queue Management Queue tasks for later processing and execute them with controlled concurrency. Tasks can be queued via the `--queue` flag on `osmedeus run` or directly via `osmedeus worker queue new`. ```bash theme={null} ▶ Full Workflow # Step 1: Queue tasks osmedeus run --queue -m recon -t example.com osmedeus run --queue -m recon -T targets.txt # Step 2: List queued tasks osmedeus worker queue list # Step 2b: List as JSON (for scripting) osmedeus worker queue list --json # Step 3: Process all queued tasks osmedeus worker queue run # Step 3b: Process with higher concurrency osmedeus worker queue run --concurrency 3 # Step 3c: Process with custom Redis URL osmedeus worker queue run --redis-url redis://localhost:6379 ▶ Shortcut # Queue and process in one step (alias for 'worker queue run') osmedeus run --queue-run osmedeus run --queue-run --concurrency 3 ▶ Direct Queue Creation # Queue a module run osmedeus worker queue new -m recon -t example.com # Queue a flow run with targets from file osmedeus worker queue new -f general -T targets.txt # Queue with parameters osmedeus worker queue new -m recon -t example.com -p 'threads=20' ``` ## 6. Server Start the Osmedeus web server that provides REST API endpoints for managing runs, workflows, and settings. ```bash theme={null} ▷ Examples # Start server with default settings osmedeus serve # Start server on custom port osmedeus serve --port 8080 # Start server without authentication (development only) osmedeus serve -A # Start server on specific host without auth osmedeus serve --host 127.0.0.1 --port 8811 -A # Start as distributed master node osmedeus serve --master # Start server without queue polling osmedeus serve --no-queue-polling ``` ## 7. Worker (Distributed Mode) Commands for managing worker nodes in distributed scanning mode. Workers connect to Redis and process tasks from the master node. ```bash theme={null} ▶ Subcommands • join - Join the distributed worker pool • status - Show worker pool status (alias: ls) • set - Update a worker field (alias, public-ip, ssh-enabled, ssh-keys-path) • eval - Evaluate a function expression with distributed hooks • queue - Manage and process queued tasks (list, new, run) ▷ Join Examples # Join using settings from osm-settings.yaml osmedeus worker join # Join using a specific Redis URL osmedeus worker join --redis-url redis://user:pass@localhost:6379/0 # Join and auto-detect public IP osmedeus worker join --get-public-ip ▷ Status # Show worker status as a table osmedeus worker status # Output worker info as JSON osmedeus worker status --json ▷ Set Worker Fields # Set an alias for a worker osmedeus worker set alias scanner-1 # Set public IP osmedeus worker set scanner-1 public-ip 203.0.113.10 # Enable SSH osmedeus worker set scanner-1 ssh-enabled true # With custom Redis URL osmedeus worker set alias prod-1 --redis-url redis://localhost:6379 ▷ Eval (one-shot with distributed hooks) # Simple function eval with distributed hooks osmedeus worker eval 'log_info("hello from worker eval")' --redis-url redis://localhost:6379 # Route a call to the master node osmedeus worker eval 'run_on_master("func", "log_info(\"routed via redis\")")' --redis-url redis://localhost:6379 # With target variable osmedeus worker eval 'log_info("hello")' -t example.com --redis-url redis://localhost:6379 # Read script from stdin echo 'run_on_master("func", "db_import_sarif(\"ws\", \"/path/f.sarif\")")' | osmedeus worker eval --stdin --redis-url redis://localhost:6379 ``` ## 8. Webhook Triggers Register runs as webhook triggers that can be invoked via HTTP requests. This allows external systems (CI/CD, monitoring tools, etc.) to trigger scans on demand. Webhook triggers require the server to have `enable_trigger_via_webhook: true` in `osm-settings.yaml`. ```bash theme={null} ▶ Setup # Step 1: Register a webhook trigger osmedeus run --as-webhook -m recon -t example.com # Register with an authentication key for security osmedeus run --as-webhook -m recon -t example.com --webhook-auth-key my-secret-key ▶ Management # List all registered webhook triggers osmedeus worker webhooks # List as JSON (for scripting) osmedeus --json worker webhooks ▶ Triggering via HTTP # Trigger a webhook (GET - no overrides) curl https://your-server/osm/api/webhook-runs//trigger # Trigger with authentication key curl https://your-server/osm/api/webhook-runs//trigger?key=my-secret-key # Trigger via POST with target override curl -X POST https://your-server/osm/api/webhook-runs//trigger \ -H 'Content-Type: application/json' \ -d '{"target": "new-target.com"}' # Trigger via POST with workflow override curl -X POST https://your-server/osm/api/webhook-runs//trigger \ -H 'Content-Type: application/json' \ -d '{"target": "example.com", "module": "subdomain"}' ▶ Configuration # Enable webhook triggering on the server osmedeus config set server.enable_trigger_via_webhook true ``` ## 9. Assets Query and list discovered assets Query and list discovered assets from the database. A shortcut for `osmedeus db ls -t assets` with first-class support for fuzzy search, source/type filtering, and asset statistics. ```bash theme={null} ▷ Examples # List all assets (default columns) osmedeus assets # Fuzzy search across asset fields osmedeus assets example.com # Filter by workspace osmedeus assets -w myworkspace # Filter by source osmedeus assets --source httpx # Filter by asset type osmedeus assets --type web # Combined filters osmedeus assets --source httpx --type web # Show asset statistics osmedeus assets --stats # Stats filtered by workspace osmedeus assets --stats -w myworkspace # With pagination osmedeus assets example.com --limit 100 # JSON output osmedeus assets example.com --json # Custom columns osmedeus assets --columns "asset_value,url,status_code" ``` | Flag | Short | Default | Description | | ------------------- | ----- | ------- | -------------------------------------------- | | `--workspace` | `-w` | `""` | Filter by workspace name | | `--source` | — | `""` | Filter by source (e.g., httpx, subfinder) | | `--type` | — | `""` | Filter by asset\_type (e.g., web, subdomain) | | `--stats` | — | `false` | Show asset statistics | | `--limit` | — | `50` | Max records to return | | `--offset` | — | `0` | Records to skip (for pagination) | | `--columns` | — | `""` | Comma-separated columns to display | | `--exclude-columns` | — | `""` | Columns to exclude from output | | `--all` | — | `false` | Show all columns including hidden ones | ## 10. Snapshot Export and import workspace snapshots as compressed ZIP archives. ```bash theme={null} ▷ Export # Export a workspace to a snapshot osmedeus snapshot export # Export with custom output path osmedeus snapshot export -o /path/to/output.zip ▷ Import # Import from a local file osmedeus snapshot import /path/to/snapshot.zip # Import from a URL osmedeus snapshot import https://example.com/snapshot.zip # Import files only (skip database import) osmedeus snapshot import /path/to/snapshot.zip --skip-db # Import without confirmation prompt osmedeus snapshot import /path/to/snapshot.zip --force ▷ List # List available snapshots osmedeus snapshot list ``` ## 11. Client (Remote Server) Interact with a remote osmedeus server via REST API. Requires environment variables for connection. ```bash theme={null} ▶ Environment Setup export OSM_REMOTE_URL="http://localhost:8002" export OSM_REMOTE_AUTH_KEY="your-api-key" ▶ Subcommands • fetch - Fetch data from server (runs, assets, vulns, etc.) • run - Create or cancel a run • exec - Execute a function remotely ▷ Fetch Examples # Fetch assets (default table) osmedeus client fetch osmedeus client fetch -t assets -w example.com # Fetch runs osmedeus client fetch --table runs osmedeus client fetch -t runs --status running # Fetch vulnerabilities with severity filter osmedeus client fetch -t vulnerabilities --severity critical # Fetch step results osmedeus client fetch -t step_results # Pagination osmedeus client fetch -t assets --limit 50 --offset 100 # JSON output osmedeus client --json fetch -t runs ▷ Run Examples # Create a flow run osmedeus client run -f basic-recon -T example.com # Create a module run osmedeus client run -m subdomain -T example.com # Cancel a run by ID osmedeus client run --cancel abc123-run-uuid # JSON output osmedeus client --json run -f recon -T example.com ▷ Exec Examples # Execute a simple function osmedeus client exec 'log_info("Hello from remote")' # With target variable osmedeus client exec -t example.com 'fileExists("{{target}}/output.txt")' # Using --script flag osmedeus client exec -s 'trim(" hello ")' # JSON output osmedeus client --json exec 'trim(" test ")' ``` ## 12. Uninstall Remove the Osmedeus installation including base folder, configuration, and optionally workspace data. ```bash theme={null} ▶ What Gets Removed • ~/osmedeus-base - Settings, workflows, binaries, data • ~/.osmedeus - Initialization marker • osmedeus binary - Removed from PATH (up to 3 locations) With --clean: • ~/workspaces-osmedeus - All scan results and workspace data ▷ Examples # Preview what will be removed (shows confirmation prompt) osmedeus uninstall # Uninstall without workspaces (keeps scan results) osmedeus uninstall --force # Full uninstall including all scan data osmedeus uninstall --force --clean ``` ## 13. Config Management Manage osmedeus configuration settings using dot notation. ```bash theme={null} ▶ Subcommands • clean - Reset configuration to defaults • set - Set a configuration value • view - View a configuration value • list - List configuration values ▷ Set Examples osmedeus config set server.port 9000 osmedeus config set server.username admin osmedeus config set server.password "d8506b99a052e797f73d1dab" osmedeus config set server.jwt.secret_signing_key "d8506b99a052e797f73d1dab" osmedeus config set scan_tactic.default 20 osmedeus config set global_vars.github_token ghp_xxx osmedeus config set notification.enabled true ▷ View Examples # Exact key lookup osmedeus config view server.port osmedeus config view server.username osmedeus config view server.jwt.secret_signing_key --redact # Wildcard pattern search (requires --force) osmedeus config view 'server.*' --force osmedeus config view 'database.*' --force osmedeus config view '*password*' --force osmedeus config view 'server.*' --force --redact ▷ List and Clean # List all config values osmedeus config list # List including secrets osmedeus config list --show-secrets # Reset config to defaults (backs up existing config first) osmedeus config clean ``` ## 14. Cloud Infrastructure Provision and manage cloud infrastructure for distributed scanning. Supports multiple providers (DigitalOcean, AWS, GCP, Linode, Azure). ```bash theme={null} ▶ Subcommands • config - Manage cloud configuration (set, list) • create - Create cloud infrastructure • list - List active cloud infrastructure • destroy - Destroy cloud infrastructure • run - Run workflow on cloud infrastructure ▷ Configuration # Set cloud provider osmedeus cloud config set defaults.provider digitalocean # Set provider credentials osmedeus cloud config set providers.digitalocean.token dop_v1_xxxx osmedeus cloud config set providers.digitalocean.region nyc1 osmedeus cloud config set providers.digitalocean.size s-2vcpu-4gb # AWS configuration osmedeus cloud config set providers.aws.access_key_id AKIAXXXX osmedeus cloud config set providers.aws.secret_access_key xxxx osmedeus cloud config set providers.aws.region us-east-1 # Set limits osmedeus cloud config set limits.max_instances 10 osmedeus cloud config set limits.max_hourly_spend 5.00 # List all cloud configuration values osmedeus cloud config list # List including secrets osmedeus cloud config list --show-secrets ▷ Infrastructure Management # Create cloud instances osmedeus cloud create --instances 3 # Create with specific provider and mode osmedeus cloud create --provider digitalocean --mode vm --instances 5 # List active infrastructure osmedeus cloud list # Destroy infrastructure by ID osmedeus cloud destroy ▷ Cloud Run # Run workflow on cloud infrastructure osmedeus cloud run -f general -t example.com --instances 3 ``` ## 15. Workflow Management List, view, search, and validate available workflows. ```bash theme={null} ▶ Subcommands • list (ls) - List available workflows (default) • show (view) - Show workflow details • validate - Validate and lint workflow(s) • install - Install workflows from source ▷ List Workflows # List all available workflows osmedeus workflow list # Search workflows by name, description, or tags osmedeus workflow ls recon osmedeus workflow ls --search subdomain # Filter by tags osmedeus workflow ls --tags recon,fast # Show tags column osmedeus workflow ls --show-tags # Show usage info osmedeus workflow ls --usage # Show workflows with parse errors (verbose mode) osmedeus workflow ls -v ▷ Show Workflow Details # Show workflow details in table format osmedeus workflow show general # Show with verbose variable descriptions osmedeus workflow show general -v # Show raw YAML with syntax highlighting osmedeus workflow show general --yaml ▷ Validate / Lint Workflows # Validate a workflow by name osmedeus workflow validate my-module # Validate a YAML file osmedeus workflow lint ./my-workflow.yaml # Validate all workflows in a folder osmedeus workflow validate /path/to/workflows/ # Stop on first failure osmedeus workflow validate . --fail-fast # CI mode (exit with error code if issues found) osmedeus workflow lint my-workflow.yaml --check # JSON output format osmedeus workflow lint my-workflow.yaml --format json # GitHub Actions annotation format osmedeus workflow lint my-workflow.yaml --format github # Disable specific lint rules osmedeus workflow validate . --disable unused-variable # Filter by minimum severity osmedeus workflow lint my-workflow.yaml --severity error ▷ Install Workflows # Install from git URL osmedeus workflow install https://github.com/user/osmedeus-workflows.git # Install from preset (uses OSM_WORKFLOW_URL or default) osmedeus workflow install --preset ``` ## 16. Update Update osmedeus to the latest version from GitHub releases. ```bash theme={null} ▷ Examples # Check for updates without installing osmedeus update --check # Update to latest version osmedeus update # Skip confirmation prompt osmedeus update --yes # Force reinstall current version osmedeus update --force # Update to a specific version osmedeus update --version v5.1.0 ``` ## 17. Version Print version and build information. ```bash theme={null} ▷ Examples # Show version info osmedeus version # JSON output osmedeus --json version ``` ## 18. Agent (ACP) Run an ACP (Agent Communication Protocol) agent interactively from the terminal. The agent command spawns an external AI coding agent as a subprocess and communicates via ACP. ```bash theme={null} ◆ Description Run an ACP agent interactively. Supports multiple agent backends. ▶ Available Agents • claude-code - Claude Code (default) — npx @zed-industries/claude-code-acp@latest • codex - OpenAI Codex — npx @zed-industries/codex-acp • opencode - OpenCode — opencode acp • gemini - Google Gemini — gemini --experimental-acp ▷ Examples # Run with default agent (claude-code) osmedeus agent "Analyze the scan results in /tmp/output" # Use a specific agent osmedeus agent --agent codex "Review this code for vulnerabilities" # List available agents osmedeus agent --list # Set working directory osmedeus agent --cwd /path/to/project "Summarize the findings" # Custom timeout (default: 30m) osmedeus agent --timeout 1h "Perform a thorough analysis" # Read message from stdin echo "Analyze this output" | osmedeus agent --stdin # Pipe with dash shorthand cat prompt.txt | osmedeus agent - ``` | Flag | Default | Description | | ----------- | ------------- | ------------------------------------------------ | | `--agent` | `claude-code` | Agent to use (see `--list` for available agents) | | `--cwd` | current dir | Working directory for the agent | | `--stdin` | `false` | Read message from stdin | | `--timeout` | `30m` | Timeout duration (e.g., 30m, 1h) | | `--list` | `false` | List available agents | ## Full Usage Examples See [Full Usage Examples](/reference/cli-references) for full CLI usage examples. # Docker Installation & Setup Source: https://docs.osmedeus.org/getting-started/docker-setup Containerized Setup and Execution Using Docker Osmedeus provides multiple Docker deployment options to suit different use cases: | Option | Best For | Image | | --------------- | ---------------------------------- | ------------------------- | | **Toolbox** | Quick start, interactive scanning | `osmedeus-toolbox:latest` | | **Production** | Standalone deployments, API server | `osmedeus:latest` | | **Distributed** | Large-scale scanning with workers | Master + Workers | ## Prerequisites * **Docker** version 20.10 or later * **Docker Compose** version 2.0 or later (included with Docker Desktop) ```bash theme={null} # Verify installation docker --version docker compose version ``` ## Quick Start with Toolbox The Toolbox image is the fastest way to get started. It includes all security tools pre-installed via the official install script. ### Using Make Targets ```bash theme={null} # Build the toolbox image make docker-toolbox # Start the container make docker-toolbox-run # Enter interactive shell make docker-toolbox-shell ``` ### Manual Docker Commands ```bash theme={null} # Build the image docker compose -f build/docker/docker-compose.toolbox.yaml build # Start the container docker compose -f build/docker/docker-compose.toolbox.yaml up -d # Enter the container docker exec -it osmedeus-toolbox bash # Run commands directly docker exec -it osmedeus-toolbox osmedeus run -m subdomain -t example.com ``` The toolbox container persists data using Docker volumes for `osmedeus-base` and `workspaces-osmedeus`, so your scan results survive container restarts. ## Running Workflows ### Interactive Mode Enter the container shell and run scans interactively: ```bash theme={null} docker exec -it osmedeus-toolbox bash # Inside the container osmedeus run -f extensive -t example.com osmedeus run -m subdomain -t example.com osmedeus workflow list ``` ### One-Off Commands Run scans without entering the container: ```bash theme={null} # Run a module docker exec -it osmedeus-toolbox osmedeus run -m subdomain -t example.com # Run a flow docker exec -it osmedeus-toolbox osmedeus run -f extensive -t example.com # List workflows docker exec -it osmedeus-toolbox osmedeus workflow list ``` ### Persisting Results with Volume Mounts For host-accessible scan results, mount local directories: ```bash theme={null} docker run -it --rm \ -v $(pwd)/workspaces:/root/workspaces-osmedeus \ -v $(pwd)/osmedeus-base:/root/osmedeus-base \ osmedeus-toolbox:latest \ osmedeus run -m subdomain -t example.com ``` ## Running the Server ### Basic Server Startup Start the REST API server inside the toolbox container: ```bash theme={null} # Start in background docker exec -d osmedeus-toolbox osmedeus serve --port 8002 --host 0.0.0.0 # Or enter container and run docker exec -it osmedeus-toolbox bash osmedeus serve --port 8002 --host 0.0.0.0 ``` ### Server with Authentication The API server uses JWT authentication by default. Configure credentials in `osm-settings.yaml`: ```yaml theme={null} server: host: "0.0.0.0" port: 8002 simple_user_map_key: admin: "your-secure-password" jwt: secret_signing_key: "your-jwt-secret-min-32-chars" expiration_minutes: 1440 ``` Start with authentication enabled: ```bash theme={null} osmedeus serve --port 8002 --host 0.0.0.0 ``` For development without authentication, use the `-A` flag: ```bash theme={null} osmedeus serve --port 8002 --host 0.0.0.0 -A ``` ## Distributed Mode with Docker Compose For large-scale scanning, use the distributed architecture with a master node coordinating multiple workers. ### Architecture ``` ┌─────────────────────────────────────────┐ │ Load Balancer │ │ (optional) │ └────────────────┬────────────────────────┘ │ ┌────────────────▼────────────────────────┐ │ Master Server │ │ - REST API (port 8002) │ │ - Task Coordinator │ │ - Web UI │ └────────────────┬────────────────────────┘ │ ┌────────────────▼────────────────────────┐ │ Redis │ │ - Message Queue │ │ - Task Distribution │ └────────────────┬────────────────────────┘ │ ┌────────────────────────────┼────────────────────────────┐ │ │ │ ┌───────▼───────┐ ┌─────────▼───────┐ ┌─────────▼───────┐ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ - Executes │ │ - Executes │ │ - Executes │ │ scans │ │ scans │ │ scans │ └───────────────┘ └─────────────────┘ └─────────────────┘ ``` ### Basic Setup The basic compose file (`docker-compose.yml`) includes Redis, master, and workers with SQLite: ```bash theme={null} # Start the distributed stack docker compose -f build/docker/docker-compose.yml up -d # Scale workers docker compose -f build/docker/docker-compose.yml up -d --scale worker=5 # View logs docker compose -f build/docker/docker-compose.yml logs -f # Stop all services docker compose -f build/docker/docker-compose.yml down ``` ### Production Setup For production deployments with PostgreSQL, use `docker-compose.production.yaml`: **Step 1: Create environment file** ```bash theme={null} cp build/docker/.env.example build/docker/.env ``` Edit `.env` with secure values: ```bash theme={null} # PostgreSQL Configuration POSTGRES_USER=osmedeus POSTGRES_PASSWORD=your_secure_postgres_password POSTGRES_DB=osmedeus # Redis (optional) # REDIS_PASSWORD=your_secure_redis_password # Server OSM_SERVER_PORT=8002 TZ=UTC # Workers WORKER_REPLICAS=2 ``` Generate secure passwords with: `openssl rand -base64 24` **Step 2: Configure application settings** The production compose file mounts `osm-settings.production.yaml`. Key settings: ```yaml theme={null} # Database - PostgreSQL for production database: db_engine: postgresql host: postgres port: 5432 username: osmedeus password: "CHANGE_ME_POSTGRES_PASSWORD" # Must match .env db_name: osmedeus ssl_mode: disable # Server server: host: "0.0.0.0" port: 8002 simple_user_map_key: admin: "CHANGE_ME_ADMIN_PASSWORD" jwt: secret_signing_key: "CHANGE_ME_JWT_SECRET_MIN_32_CHARS" expiration_minutes: 1440 # Redis (required for distributed mode) redis: host: redis port: 6379 password: "" # Set if REDIS_PASSWORD is configured in .env ``` **Step 3: Start the production stack** ```bash theme={null} # Start infrastructure first docker compose -f build/docker/docker-compose.production.yaml up -d postgres redis # Wait for healthy status (about 10-15 seconds) docker compose -f build/docker/docker-compose.production.yaml ps # Start the application docker compose -f build/docker/docker-compose.production.yaml up -d # Scale workers for parallel scanning docker compose -f build/docker/docker-compose.production.yaml up -d --scale worker=5 ``` ### Environment Variables | Variable | Description | Default | | ------------------- | --------------------------- | ------------ | | `POSTGRES_USER` | PostgreSQL username | `osmedeus` | | `POSTGRES_PASSWORD` | PostgreSQL password | **Required** | | `POSTGRES_DB` | PostgreSQL database name | `osmedeus` | | `REDIS_PASSWORD` | Redis password (optional) | - | | `OSM_SERVER_PORT` | External API port | `8002` | | `TZ` | Container timezone | `UTC` | | `WORKER_REPLICAS` | Number of worker containers | `2` | ### Scaling Workers Workers can be scaled dynamically based on workload: ```bash theme={null} # Scale to 10 workers docker compose -f build/docker/docker-compose.production.yaml up -d --scale worker=10 # Scale back down docker compose -f build/docker/docker-compose.production.yaml up -d --scale worker=2 ``` Each worker has resource limits (configurable in compose file): * CPU: 2 cores (limit), 0.5 cores (reservation) * Memory: 2GB (limit), 512MB (reservation) ## Building Custom Images ### Production Image The production Dockerfile (`build/docker/Dockerfile`) creates a minimal image: ```bash theme={null} # Build with make make docker-build # Or build directly docker build -t osmedeus:latest \ -f build/docker/Dockerfile \ --build-arg VERSION=5.0.0 \ . ``` ### Development Image For development with hot-reloading: ```bash theme={null} docker build -t osmedeus:dev -f build/docker/Dockerfile.dev . docker run -it --rm \ -v $(pwd):/app \ -p 8002:8002 \ osmedeus:dev ``` The development image includes: * Full Go toolchain * Air for hot-reloading * Vim for editing ### Toolbox Image Build the full-featured toolbox with all tools: ```bash theme={null} docker build -t osmedeus-toolbox:latest \ -f build/docker/Dockerfile.toolbox \ . ``` ## Volume Management Osmedeus uses two primary data directories: | Volume | Container Path | Purpose | | --------------- | --------------------------- | ---------------------------------- | | `osmedeus-data` | `/root/osmedeus-base` | Configuration, workflows, database | | `workspaces` | `/root/workspaces-osmedeus` | Scan results and artifacts | ### Backing Up Volumes ```bash theme={null} # List volumes docker volume ls | grep osmedeus # Backup workspaces docker run --rm \ -v osmedeus-workspaces:/data \ -v $(pwd):/backup \ alpine tar czf /backup/workspaces-backup.tar.gz -C /data . # Restore workspaces docker run --rm \ -v osmedeus-workspaces:/data \ -v $(pwd):/backup \ alpine tar xzf /backup/workspaces-backup.tar.gz -C /data ``` ### Removing Volumes ```bash theme={null} # Remove all osmedeus volumes (WARNING: deletes all data) docker compose -f build/docker/docker-compose.production.yaml down -v ``` ## Troubleshooting ### Container Won't Start ```bash theme={null} # Check container logs docker logs osmedeus-toolbox # Check compose logs docker compose -f build/docker/docker-compose.yml logs -f master ``` ### Permission Issues If you encounter permission issues with mounted volumes: ```bash theme={null} # Run container as root (default) docker exec -u root -it osmedeus-toolbox bash # Or fix host directory permissions sudo chown -R $(id -u):$(id -g) ./workspaces ``` ### Database Connection Errors For PostgreSQL connection issues: ```bash theme={null} # Check PostgreSQL is healthy docker compose -f build/docker/docker-compose.production.yaml ps postgres # Test connection from master docker exec osmedeus-server psql -h postgres -U osmedeus -d osmedeus -c "SELECT 1" ``` ### Redis Connection Errors ```bash theme={null} # Check Redis is healthy docker exec osmedeus-redis redis-cli ping # Test with password docker exec osmedeus-redis redis-cli -a your_password ping ``` ### Health Check Failures ```bash theme={null} # Check master health endpoint curl http://localhost:8002/health # View health check logs docker inspect --format='{{json .State.Health}}' osmedeus-server | jq ``` ### Out of Memory If workers run out of memory, adjust resource limits in the compose file: ```yaml theme={null} worker: deploy: resources: limits: memory: 4G reservations: memory: 1G ``` # Web UI Source: https://docs.osmedeus.org/getting-started/web-ui Server and Web UI for Osmedeus This document describes the web UI for Osmedeus which is the main entry point for running workflows and modules and some other utilities. The Osmedeus server also acts as an event receiver, processing events from other runs and handling scheduled scans. See [Event-Driven](/advanced/event-driven) for more details. Start the web UI by running the following command: ```bash theme={null} osmedeus serve ``` then open your browser and go to `https://localhost:8002`. ## How to Login into Web UI web-ui-login Your default password is in `$HOME/osmedeus-base/osm-settings.yaml` You can view the default credentials by running the following command as it is auto-generated: ```bash theme={null} osmedeus config view server.username osmedeus config view server.password ``` Many API endpoints are intentionally designed to execute code on your machine, so ensure your web UI and API endpoints are protected with strong credentials. See [security-warning](/others/security-warning) for more details You can also change the default password and set API key authentication by running the following command: ```bash theme={null} osmedeus config set server.username "osmedeus" osmedeus config set server.password "$(openssl rand -hex 12)" # api key auth requires a jwt secret signing key osmedeus config set server.jwt.secret_signing_key "$(openssl rand -hex 32)" osmedeus config set server.enabled_auth_api true osmedeus config set server.auth_api_key "$(openssl rand -hex 24)" ## API Key Authentication Settings # server: # enabled_auth_api: true # auth_api_key: "your-secure-api-key" ``` *** ## All Web UI Pages The web UI consists of the following pages that allow you to view and manage your assets, workspaces, vulnerabilities, and other utilities. ### 1. Assets and Workspace This is the main page of the web UI where you can view and manage your assets, workspaces, vulnerabilities, and artifacts which will be generated by the workflows. web-ui-workspace web-ui-assets web-ui-vuln web-ui-assets web-ui-vuln You also have the option to enhance the UI, displaying artifacts in a cleaner, syntax-highlighted layout—ideal for reviewing markdown or HTML reports. web-ui-artifact-list web-ui-artifact-details ### 2. Start New Scan (Simple Run & Scheduled) This is where you can start a new scan by selecting the workflow and the target asset with all the extra parameters and scheduling options. web-ui-new After you start a new scan, you can view the progress and results in the list of scans. web-ui-list-scan ### 3. Settings, Install Registry This is where you can configure the settings and install the registry for Osmedeus. web-ui-install-registry web-ui-settings ### 4. Utilities Functions & Scheduling This is where you can schedule the workflows to run at specific times or intervals, and also use the LLM chat to get help with your workflows. web-ui-schedule web-ui-llm-chat web-ui-utility-functions ## Workflow Visualization and Editor Visualization and editor for workflows in the web UI through the beautifl representation of the workflow via xyflow. web-ui-workflow1 web-ui-workflow2 web-ui-workflow3 web-ui-workflow4 web-ui-workflow4 ## Communication with Web UI Once you have logged in, you can use the web UI to run workflows, view the results, and manage your Osmedeus installation. ```bash Request theme={null} curl --request POST \ --url https://localhost:8002/osm/api/runs \ --header 'Authorization: Bearer $TOKEN' \ --header 'Content-Type: application/json' \ --data '{"flow": "general", "target": "example.com", "distributed": true}' ``` # Introduction Source: https://docs.osmedeus.org/index Welcome to the Osmedeus Documentation Introduction Banner [Osmedeus](https://github.com/j3ssie/osmedeus) is a security focused declarative orchestration engine that simplifies complex workflow automation into auditable YAML definitions, complete with encrypted data handling, secure credential management, and sandboxed execution. Built for both beginners and experts, it delivers powerful, composable automation without sacrificing the integrity and safety of your infrastructure. ## Key Features * **Declarative YAML Workflows** - Define pipelines with hooks, decision routing, module exclusion, and conditional branching across multiple runners (host, Docker, SSH) * **Distributed Execution** - Redis-based master-worker pattern with queue system, webhook triggers, and file sync across workers * **Rich Function Library** - 80+ utility functions including nmap integration, tmux sessions, SSH execution, TypeScript/Python scripting, SARIF parsing, and CDN/WAF classification * **Event-Driven Scheduling** - Cron, file-watch, and event triggers with filtering, deduplication, and delayed task queues * **Agentic LLM Steps** - Tool-calling agent loops with sub-agent orchestration, memory management, and structured output * **Cloud Infrastructure** - Provision and run scans across DigitalOcean, AWS, GCP, Linode, and Azure with cost controls and automatic cleanup * **Rich CLI Interface** - Interactive database queries, bulk function evaluation, workflow linting, progress bars, and comprehensive usage examples * **REST API & Web UI** - Full API server with webhook triggers, database queries, and embedded dashboard for visualization Hall of fame in light mode Hall of fame in dark mode ## Getting Started CLI Run preview Web UI Assets preview Jump right in and run your first Osmedeus workflow in minutes. *** ## Advanced Installation and Configuration Detailed instructions for installing Osmedeus on various platforms. Configure the engine, runners, and environment variables. Deploy Osmedeus in a distributed environment or production setup. Resources for developers contributing to or extending Osmedeus. ## Understanding Osmedeus ### Core Concepts | Page | Description | | ---------------------------------------- | ----------------------------------------------------------------------------------- | | [Architecture](concepts/architecture.md) | Layered architecture and data flow | | [Workflows](concepts/workflows.md) | Module vs Flow, execution lifecycle | | [Templates](concepts/templates.md) | Variable interpolation and built-in variables | | [Runners](concepts/runners.md) | Host, Docker, SSH execution environments | | [Functions](concepts/functions.md) | JavaScript utility functions that bind to the core engine for use in workflow steps | ### Advanced Topics | Page | Description | | ------------------------------------------------ | ------------------------------------ | | [Distributed Execution](advanced/distributed.md) | Master-worker architecture | | [Scheduling](advanced/scheduling.md) | Cron, event, and file-watch triggers | | [LLM Integration](advanced/llm.md) | AI-powered workflow steps | | [Snapshots](advanced/snapshots.md) | Workspace export and import | *** ### Workflows | Page | Description | | ----------------------------------------- | ------------------------------------------ | | [Overview](workflows/overview.md) | YAML structure and workflow kinds | | [Step Types](workflows/step-types.md) | All 8 step types with examples | | [Flows](workflows/flows.md) | Module orchestration and dependencies | | [Variables](workflows/variables.md) | Parameters, exports, variable propagation | | [Control Flow](workflows/control-flow.md) | Conditions, handlers, and decision routing | ### Extending Osmedeus | Page | Description | | ------------------------------------------- | -------------------------- | | [Step Types](extending/step-types.md) | Add custom step executors | | [Runners](extending/runners.md) | Implement new runner types | | [Functions](extending/functions.md) | Register utility functions | | [CLI Commands](extending/cli-commands.md) | Add new CLI commands | | [API Endpoints](extending/api-endpoints.md) | Add new REST endpoints | ## Reference | Page | Description | | ----------------------------------------------- | -------------------- | | [Workflow Schema](reference/workflow-schema.md) | Complete YAML schema | | [Variables](reference/variables.md) | Built-in variables | | [Types](reference/types.md) | Go type definitions | ## Full Feature List * **Declarative YAML Workflows** - Define reconnaissance pipelines using simple, readable YAML syntax * **Multiple Runners** - Execute on local host, Docker containers, or remote machines via SSH * **Event-Driven Triggers** - Cron scheduling, file watching, and event-based workflow triggers with deduplication and filter functions * **Template Engine** - Powerful variable interpolation with built-in and custom variables * **Utility Functions** - Rich function library with event generation, bulk processing, and JSON operations * **REST API Server** - Manage, trigger, and cancel workflows programmatically * **Distributed Execution** - Scale with Redis-based master-worker pattern for parallel scanning (workers identified as `wosm-`) * **Notifications** - Telegram bot and webhook integrations * **Cloud Storage** - S3-compatible storage for artifact management * **LLM Integration** - AI-powered workflow steps with chat completions, embeddings, and agentic tool-calling loops * **Agent Step Type** - Agentic LLM execution with tool calling, sub-agents, and memory management * **SAST Integration** - SARIF parsing for Semgrep, Trivy, Kingfisher, Bearer with database import and markdown reporting * **Language Detection** - Auto-detect dominant programming language of source repositories (26+ languages) * **Preset Installation** - Reproducible deployments from curated preset repositories * **Workflow Hooks** - Pre/post scan steps via `hooks` field for setup and cleanup * **Queue System** - Delayed task execution with database and Redis polling, configurable concurrency * **Nmap Integration** - Port scanning with automatic XML/gnmap to JSONL conversion and database import * **Tmux Sessions** - Background process management via tmux (create, capture, send, kill sessions) * **SSH & Sync** - Remote execution and file synchronization across distributed workers * **TypeScript Execution** - Run inline TypeScript or TS files via Bun runtime * **Webhook Triggers** - Trigger workflow runs via unauthenticated webhook URLs * **CDN/WAF Classification** - Automatic asset classification from httpx data (CDN, cloud, WAF) * **Module Exclusion** - Exclude modules from flows by exact name or fuzzy substring matching * **Cloud Infrastructure** - Provision and manage cloud instances across multiple providers # FAQs Source: https://docs.osmedeus.org/others/faq FAQs and Common Errors A collection of frequently asked questions along with explanations of common errors and how to resolve them. ## 1. General Osmedeus is a workflow engine for security automation. It executes YAML-defined workflows with support for multiple execution environments (host, Docker, SSH), scheduling, and distributed scanning. The Osmedeus core engine is lightweight and can run anywhere with almost any specs.
However, if you plan to use it for **reconnaissance** (which is the main use case), it is recommended to use a modern Linux, macOS, or Windows system with WSL.
Since running reconnaissance generates heavy network traffic, it is also recommended to run Osmedeus in a cloud environment, **such as a VM, Compute Engine, or EC2**, to achieve the best performance. ```bash theme={null} # Build from source make build # Install to $GOBIN make install # Install security tools osmedeus install binary --all ``` Yes of course. Osmedeus has built-in support for LLMs and you can use it in your workflow to do things like generating recon reports, writing custom scripts, or even building your own agentic workflow. You can check out the [LLM Workflow Example](/advanced/llm/) to see how it works. Be aware that using LLMs may require you to have API keys for the LLM provider and may incur additional costs based on your usage. Always monitor your usage and costs when using LLMs in your workflows. Since Osmedeus is an orchestration framework, you can leverage it to coordinate your own custom AI/LLM tools including integrations like Claude Code or OpenCode directly within your YAML workflows. For instance, you could design a custom agent that invokes multiple tools as part of a defined pipeline and seamlessly plug it into your workflow. The flexibility is virtually unlimited. Yes, I've built the [osmedeus-expert](https://github.com/osmedeus/osmedeus-skills) skill at [github.com/osmedeus/osmedeus-skills](https://github.com/osmedeus/osmedeus-skills) and you can use it in your agentic tool to writing YAML workflows, running CLI commands, and configuring advanced features. *** ## 2. Binary Installation Osmedeus is a standalone Golang binary and works perfectly fine on its own. However, when using Osmedeus to run YAML workflows for security automation, it often needs to call external tools like `httpx, nuclei, ffuf, etc`. These tools must be installed and available on your system for those workflows to function properly. Not all binaries listed in the registry are required for every workflow. Your scans may still function correctly even if some tools are missing. No. Installing all tools is completely optional. The registry includes additional tools that are commonly used in YAML workflows, but you only need the ones required for your specific workflow. Installing everything is not necessary for running a basic workflow. Like I said above, not all binaries listed in the registry are required for every workflow. Your scans may still function correctly even if some tools are missing. If you would like the ideal setup then I recommend using Docker to run Osmedeus and its workflows. This ensures that all dependencies are met and eliminates any compatibility issues. See the [Docker Setup](/getting-started/docker-setup) for more details. *** ## 3. Scan Execution & Scanning Results ```bash theme={null} # Run a flow workflow osmedeus run -f general -t example.com # Run a module workflow osmedeus run -m vulnerability-scan -t example.com ``` ```bash theme={null} # From command line osmedeus run -f fast -t target1.com -t target2.com # From file osmedeus run -f fast -T targets.txt -c 5 ``` ```bash theme={null} osmedeus run -m port-scan -t example.com --timeout 2h ``` Results are stored in workspaces at `~/workspaces-osmedeus//`. All you need to do is follow [**this guide to setup the token**](/getting-started/basic-setup/) All you need to do is follow [**this guide to setup notification**](/advanced/notification-and-cdn/) You can Join **`https://discord.gg/mtQG2FQsYA`** to see if anyone can help. I might answer from time to time but I couldn't promise to answer every single of them. Nope, natively it doesn't support proxy. But since the design of the tool is running other 3rd party tools and a lot of them don't support proxy by default. I've already considered proxychains but it makes it extremely slow and breaks a lot of things. It will stay there because it got a sudo password prompt. Some special tools require *root* permission to run like **nmap**. Make sure you allow **nmap** can be run without sudo password prompt. It's probably because the thing you put in was really big. Think about trying to run the content discovery against **2000 different hosts**. That's why it takes a long time. Again it very much depends on your target. Osmedeus really shines on large scope targets, not the single intentionally vulnerable web app. Just scan some random VDP then you will see the result. The reason it won't find any vulnerability on the intentionally vulnerable app is the **vulnscan** module won't support it. But you're always welcome to customize the workflow to do so. Yes, just follow [**this guide**](/advanced/writing-your-first-workflow/) to add it to your workflow. 1. Read the flow and module files to determine what a step actually runs 2. Seriously, read the flow and module files. 3. Remember that you were warned twice about reading the flow and module files. 4. Search for the tool command in the workflow folder to confirm whether it is used or not (e.g: `rg -F 'nuclei' ~/osmedeus-base/workflows/`) Please refer to [**this page**](/getting-started/web-ui/) to start a web server and get credentials. You may need to run this command `osmedeus config view server.password` The simplest way to do it is running the process under `https://tmuxcheatsheet.com/` . Other than that you can setup a service to run the osmedeus web server as a background process. 1. Read the vulnerability X description. 2. Seriously, read the vulnerability X description. 3. Remember that you were warned twice about reading the vulnerability X description. 4. Search for that vulnerability X name. 5. Manually verify the vulnerability X. 6. Still no results? maybe `https://letmegooglethat.com/?q=what+is+a+vulnerability+X` can help you. It is often the case that the availability of a subdomain found during a scan may not be the same when you attempt to manually verify it. This depends on the target and can vary. Yes, it's normal for certain commands to exhibit expected exit statuses, as they may succeed under specific conditions. However, if you're confident that the raw bash command should succeed but is failing, please try copying the raw bash command and investigate why it's encountering issues. You can run `osmedeus workflow ls` or `osmedeus workflow show --verbose` to see the description and that would fit to the scan This is likely due to the fact that the workflow you executed did not generate any assets. You can verify this by checking the workspace directory located at `~/workspaces-osmedeus//` to see if any files were created. It is also because the workflow doesn't use any database utility function to save the assets into the database. You can check the workflow file to see if it uses any database utility functions like `db_import_asset`. You can also see the full list of database related function at `osmedeus func ls db --example` This is likely due to the fact that the workflow you executed did not generate any assets. You can verify this by checking the workspace directory located at `~/workspaces-osmedeus//` to see if any files were created. It is also because the workflow doesn't use any notification utility function to save the assets into the notification. You can check the workflow file to see if it uses any notification utility functions like `notify_telegram`. You can also see the full list of notification related function at `osmedeus func ls noti --example` *** ## 4. Workflows * **Module**: A single workflow unit containing steps that execute sequentially * **Flow**: Orchestrates multiple modules, allowing parallel execution and dependencies between modules Workflows are stored in `~/osmedeus-base/workflows/`: * `flows/` - Flow workflows * `modules/` - Module workflows Create a YAML file in the workflows directory: ```yaml theme={null} name: my-workflow kind: module description: My custom workflow params: - name: target required: true steps: - name: scan-target type: bash command: nmap {{target}} ``` | Type | Description | | ---------------- | --------------------------------------- | | `bash` | Execute shell commands | | `function` | Execute JavaScript utility functions | | `parallel-steps` | Run steps concurrently | | `foreach` | Iterate over items | | `remote-bash` | Execute in Docker or via SSH | | `http` | Make HTTP requests | | `llm` | Execute LLM API calls | | `agent` | Agentic LLM execution with tool calling | *** ## 5. API & Server ```bash theme={null} osmedeus server ``` The server starts on port 8002 by default. ```bash theme={null} $ osmedeus server 2026-02-16T22:59:11+07:00 ERROR Failed to create server {"error": "failed to run database migrations: failed to create index: SQL logic error: no such column: webhook_uuid (1)"} Error: failed to run database migrations: failed to create index: SQL logic error: no such column: ... (1) ``` Error like this means that the database schema is outdated and the server cannot start. To fix this, you can run `osmedeus db clean --force` to clean up the database and then start the server again. This will reset your database, so make sure to backup any important data before running the command. ```bash theme={null} # Get a JWT token curl -X POST http://localhost:8002/osm/api/login \ -H "Content-Type: application/json" \ -d '{"username": "osmedeus", "password": "admin"}' # Use the token curl http://localhost:8002/osm/api/workflows \ -H "Authorization: Bearer " ``` ```bash theme={null} osmedeus server --no-auth ``` Yes, enable API key authentication in the server configuration. Then use the `X-API-Key` header instead of `Authorization: Bearer`. *** ## 6. Scheduling ```bash theme={null} # Via CLI (creates a cron schedule) osmedeus run -f subdomain-enum -t example.com --schedule "0 2 * * *" # Via API curl -X POST http://localhost:8002/osm/api/schedules \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "daily-scan", "workflow_name": "subdomain-enum", "target": "example.com", "schedule": "0 2 * * *" }' ``` Standard 5-field cron: `minute hour day-of-month month day-of-week` Examples: * `0 2 * * *` - Daily at 2 AM * `0 0 * * 0` - Weekly on Sunday * `*/30 * * * *` - Every 30 minutes *** ## 7. Runners | Runner | Description | | -------- | ---------------------------------- | | `host` | Execute on local machine (default) | | `docker` | Execute in Docker containers | | `ssh` | Execute on remote machines via SSH | ```bash theme={null} osmedeus run -m port-scan -t example.com --runner docker --docker-image osmedeus/osmedeus:latest ``` ```bash theme={null} osmedeus run -m port-scan -t example.com --runner ssh --ssh-host worker.example.com ``` *** ## 8. Distributed Mode Start the master: ```bash theme={null} osmedeus server --master ``` Join workers: ```bash theme={null} osmedeus worker join --master http://master:8002 ``` ```bash theme={null} curl -X POST http://localhost:8002/osm/api/tasks \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "workflow_name": "subdomain-enum", "target": "example.com" }' ``` *** ## 9. Troubleshooting ```bash theme={null} osmedeus install binary --all --check ``` ```bash theme={null} # Install specific tools osmedeus install binary --name nuclei --name httpx # Install all tools osmedeus install binary --all ``` Logs are stored in the workspace: ```bash theme={null} cat ~/osmedeus-base/workspaces//log/execution.log ``` ```bash theme={null} osmedeus snapshot export ``` ```bash theme={null} osmedeus snapshot import snapshot.zip ``` *** ## 10. Configuration `~/osmedeus-base/osm-settings.yaml` Edit `osm-settings.yaml`: ```yaml theme={null} server: port: 9000 ``` Or use the `--port` flag: ```bash theme={null} osmedeus server --port 9000 ``` Edit `osm-settings.yaml`: ```yaml theme={null} database: db_engine: sqlite3 # or postgres host: localhost port: 5432 name: osmedeus username: user password: pass ``` ## 11. Clean up & Uninstall just run the command below to clean up workspace and database and generate the default osmedeus config ```bash theme={null} rm -rf ~/osmedeus-base ~/workspaces-osmedeus osmedeus install base --preset ``` just run the command below to clean up workspace and database and generate the default osmedeus config ```bash theme={null} rm -rf ~/workspaces-osmedeus osmedeus db clean --force ``` just run the command below ```bash theme={null} rm -rf ~/.osmedeus ~/osmedeus-base ~/workspaces-osmedeus rm -rf $(which osmedeus) ``` # Premium Edition Source: https://docs.osmedeus.org/others/premium-edition Exclusive features and benefits available in the Osmedeus Premium Edition Make a donation through any available method [**Sponsor**](/sponsor) to receive the premium edition installation link Unleash the full potential of Osmedeus with the **Premium Edition**. Get access to exclusive workflows, advanced reporting, and premium tool integrations designed for serious security professionals. **Installation** - Premium users receive a custom one-liner install command in Discord or Patreon. Run it on any supported system to get started: ```bash theme={null} curl -fsSL https:///premium-osmedeus-install.sh | bash ``` Replace `` with the domain provided in your premium access credentials. *** ## Features **Deeper scans. Better findings. Real results.** ### Premium Workflows Optimized scanning workflows that chain multiple tools together for maximum coverage and deeper findings. These workflows are battle-tested against real-world targets and continuously updated with the latest techniques. ### Advanced Reporting Generate detailed, actionable security reports with custom templates. Export findings in formats ready for stakeholders, compliance teams, or bug bounty submissions. ### Exclusive Tool Integrations Access private and premium security tools not available in the community edition, including specialized crawlers and scanners that extend your reconnaissance capabilities. ### Extended Attack Surface Deep subdomain enumeration, comprehensive port scanning, and aggressive web crawling configurations that uncover what others miss. Designed for maximum coverage on large-scope targets. *** ## Frequently Asked Questions Premium Osmedeus includes distributed scanning across multiple machines, exclusive premium workflows, exclusive tools, cloud provider integration, advanced reporting templates, continuous monitoring, and priority support from the development team. Osmedeus Premium runs on Linux (Ubuntu/Debian recommended). You need at least 2 CPU cores, 4 GB RAM, and 20 GB of disk space. The installer will handle all dependencies automatically. Run the same install command again. The installer detects existing installations and performs an in-place upgrade, preserving your configuration and scan data. Premium users get direct access to the development team. Reach out via the dedicated support channel provided after installation, or open an issue on the [GitHub repository](https://github.com/j3ssie/osmedeus) for community support. # Security Warning Source: https://docs.osmedeus.org/others/security-warning Important Security Notice and Disclaimer **Important:** Osmedeus is a powerful security automation tool designed to execute code on your machine. Please read this document carefully before using Osmedeus in any environment. ## Overview Osmedeus is intentionally designed as a workflow execution engine that runs arbitrary commands and scripts. This design is fundamental to its purpose as a security automation tool. However, this power comes with inherent security risks that users must understand and mitigate. This document outlines the security considerations, potential risks, and best practices for safely using Osmedeus. *** ## Security Considerations ### 1. Web UI and API Server The Osmedeus Web UI and REST API provide interfaces for: * Creating and executing new scans * Running utility functions * Managing workflows and schedules * Accessing scan results and artifacts **Risks:** * Unauthorized access could allow attackers to execute arbitrary commands on your system * Exposed APIs without authentication can be exploited for remote code execution * Default credentials pose a significant security risk **Recommendations:** * Always use strong, unique credentials for API authentication. Use the following commands to set secure random credentials: ```bash theme={null} osmedeus config set server.password "$(openssl rand -hex 12)" osmedeus config set server.jwt.secret_signing_key "$(openssl rand -hex 32)" osmedeus config set server.auth_api_key "$(openssl rand -hex 24)" ``` * Never expose the API server to the public internet without proper authentication * Use the `--no-auth` flag only in isolated development environments * Consider using API keys with limited permissions * Deploy behind a reverse proxy with TLS encryption * Implement network-level access controls (firewall rules, VPN) ### 2. YAML Workflow Files YAML workflow files are the core of Osmedeus automation. They can contain: * Shell commands (`bash` steps) * JavaScript function calls (`function` steps) * Remote execution commands (`remote-bash` steps) * HTTP requests (`http` steps) **Risks:** * Malicious workflows can execute arbitrary commands with the privileges of the Osmedeus process * Third-party workflows may contain hidden malicious code * Workflows can access the filesystem, network, and other system resources **Recommendations:** * **Never run untrusted or unverified workflow files** * Always review workflow YAML files before execution * Use `osmedeus workflow validate ` to check workflow syntax * Store workflows in version-controlled repositories * Implement workflow signing or checksums for verification * Run Osmedeus with minimal required privileges This is similar to other workflow engines such as Apache Airflow, Argo Workflows, GitHub Actions, and Jenkins. Allowing users to execute arbitrary workflows is inherent to their design. ### 3. External Binary Installation The `osmedeus install` command downloads and installs binaries from external sources: * Security tools (nuclei, httpx, subfinder, etc.) * Workflow files from remote repositories **Risks:** * Downloaded binaries could be compromised or malicious * Man-in-the-middle attacks during downloads * Supply chain attacks on upstream tool repositories **Recommendations:** * Only install binaries from trusted sources * Verify checksums when available * Consider using the Nix-based installation for reproducible builds * Review the binary registry before installing new tools * Keep installed tools updated to patch known vulnerabilities ### 4. Database and Storage Osmedeus stores scan results, credentials, and configuration data: **Risks:** * Sensitive data exposure through database access * Unencrypted storage of credentials * Backup file exposure **Recommendations:** * Secure database access with strong authentication * Encrypt sensitive configuration values * Implement proper backup encryption * Regularly audit stored data for sensitive information * Use PostgreSQL with TLS for production deployments *** ## Best Practices Summary | Area | Recommendation | | -------------- | ------------------------------------------------------------- | | Authentication | Use strong credentials, enable API keys, rotate regularly | | Network | Use TLS, firewall rules, VPN for remote access | | Workflows | Review before execution, use version control, validate syntax | | Binaries | Verify sources, check signatures, use Nix when possible | | Privileges | Run with minimal permissions, use dedicated service accounts | | Monitoring | Enable logging, audit access, monitor for anomalies | | Updates | Keep Osmedeus and tools updated with security patches | *** ## Disclaimer **Osmedeus is for authorized security testing only.** Unauthorized use may violate laws in your jurisdiction. By using Osmedeus, you acknowledge: * **Authorization Required** — You must have explicit permission before scanning any target * **User Responsibility** — You are solely responsible for legal compliance and any consequences from use * **No Warranty** — Provided "AS IS" without warranty; authors are not liable for damages or legal issues * **Code Execution** — This tool intentionally executes code by design; review all workflows before running * **Third-Party Tools** — You must comply with the licenses and terms of all integrated tools **Limitation of Liability:** The authors shall not be liable for any claims, damages, or liability arising from use of this software. The user assumes all responsibility and risk. *** ## Reporting Security Issues If you discover a security vulnerability in Osmedeus: 1. **Do not** disclose it publicly until it has been addressed 2. Report the issue through [GitHub Security Advisories](https://github.com/j3ssie/osmedeus/security/advisories) 3. Provide detailed information to help reproduce and fix the issue 4. Allow reasonable time for the issue to be addressed before disclosure # Quickstart Source: https://docs.osmedeus.org/quickstart Get up and running with Osmedeus in 5 minutes ## Step 1: Install the latest version of Osmedeus ```bash theme={null} curl -fsSL https://www.osmedeus.org/install.sh | bash ``` ```bash theme={null} brew install osmedeus/tap/osmedeus ``` ```bash theme={null} curl -fsSL https://www.osmedeus.org/nightly-install.sh | bash ``` ```bash theme={null} git clone https://github.com/osmedeus/osmedeus.git cd osmedeus make build ``` Osmedeus only supports Linux and macOS natively. Windows users please use WSL or See [Docker Setup here](/getting-started/docker-setup) ## Step 2: Install the default workflows ```bash theme={null} osmedeus install base --preset ``` ```bash theme={null} osmedeus install workflow https://github.com/osmedeus/osmedeus-workflow.git ``` ## Step 3: Validate the installation ```bash theme={null} osmedeus health ``` Check out the [FAQs and Common Errors](/others/faq) if you encounter any issues during the installation. ## Step 4: Run your first workflow ```bash theme={null} osmedeus run -m subdomain -t example.com ``` CLI Run preview Web UI Assets preview # Type Reference Source: https://docs.osmedeus.org/reference/types Complete reference for all type constants used in Osmedeus workflows # Type Reference This document provides a comprehensive reference for all type constants used in Osmedeus workflows. ## WorkflowKind The `WorkflowKind` type defines the category of a workflow. Each workflow must declare exactly one kind. | Constant | Value | Description | | -------------- | ------------ | ------------------------------------------------- | | `KindModule` | `"module"` | Single unit workflow containing steps | | `KindFlow` | `"flow"` | Orchestrates multiple modules | | `KindFragment` | `"fragment"` | Reusable step collection for embedding in modules | ### Helper Methods * `IsModule()` - Returns true if workflow is a module * `IsFlow()` - Returns true if workflow is a flow * `IsFragment()` - Returns true if workflow is a fragment ## StepType The `StepType` type defines the execution behavior of a step within a module or fragment. | Constant | Value | Description | | ---------------------- | ------------------ | -------------------------------------- | | `StepTypeBash` | `"bash"` | Execute shell commands locally | | `StepTypeFunction` | `"function"` | Execute utility functions | | `StepTypeParallel` | `"parallel-steps"` | Execute steps in parallel | | `StepTypeForeach` | `"foreach"` | Iterate over input lines | | `StepTypeRemoteBash` | `"remote-bash"` | Execute commands remotely (Docker/SSH) | | `StepTypeHTTP` | `"http"` | Make HTTP requests | | `StepTypeLLM` | `"llm"` | Interact with LLM APIs | | `StepTypeFragmentStep` | `"fragment-step"` | Execute a fragment inline | ### Helper Methods * `IsBashStep()` - Returns true if step is bash type * `IsFunctionStep()` - Returns true if step is function type * `IsParallelStep()` - Returns true if step is parallel-steps type * `IsForeachStep()` - Returns true if step is foreach type * `IsRemoteBashStep()` - Returns true if step is remote-bash type * `IsHTTPStep()` - Returns true if step is http type * `IsLLMStep()` - Returns true if step is llm type * `IsFragmentStep()` - Returns true if step is fragment-step type ## TriggerType The `TriggerType` type defines how workflows can be triggered. | Constant | Value | Description | | --------------- | ---------- | --------------------------------------- | | `TriggerCron` | `"cron"` | Scheduled execution via cron expression | | `TriggerEvent` | `"event"` | Triggered by system events | | `TriggerWatch` | `"watch"` | Triggered by file system changes | | `TriggerManual` | `"manual"` | Manual CLI execution | ## RunnerType The `RunnerType` type defines the execution environment for steps. | Constant | Value | Description | | ------------------ | ---------- | ---------------------------------- | | `RunnerTypeHost` | `"host"` | Execute on local machine (default) | | `RunnerTypeDocker` | `"docker"` | Execute in Docker container | | `RunnerTypeSSH` | `"ssh"` | Execute on remote machine via SSH | ## VariableType The `VariableType` type defines input validation for workflow parameters. | Constant | Value | Description | | ------------------ | ------------- | -------------------- | | `VarTypeDomain` | `"domain"` | Valid domain name | | `VarTypeSubdomain` | `"subdomain"` | Valid subdomain | | `VarTypeURL` | `"url"` | Valid URL | | `VarTypeCIDR` | `"cidr"` | Valid CIDR notation | | `VarTypePath` | `"path"` | File system path | | `VarTypeFile` | `"file"` | Existing file path | | `VarTypeFolder` | `"folder"` | Existing folder path | | `VarTypeNumber` | `"number"` | Numeric value | | `VarTypeString` | `"string"` | Any string value | | `VarTypeRepo` | `"repo"` | Git repository URL | ## TargetType The `TargetType` type defines target input classification. | Constant | Value | Description | | --------------------- | ------------- | ----------------------- | | `TargetTypeDomain` | `"domain"` | Domain name target | | `TargetTypeSubdomain` | `"subdomain"` | Subdomain target | | `TargetTypeURL` | `"url"` | URL target | | `TargetTypeCIDR` | `"cidr"` | CIDR range target | | `TargetTypeRepo` | `"repo"` | Git repository target | | `TargetTypePath` | `"path"` | File system path target | | `TargetTypeFile` | `"file"` | File target | | `TargetTypeFolder` | `"folder"` | Folder target | | `TargetTypeNumber` | `"number"` | Numeric target | | `TargetTypeString` | `"string"` | String target | ## ActionType The `ActionType` type defines handlers for `on_success` and `on_error` events. | Constant | Value | Description | | ---------------- | ------------ | ------------------------- | | `ActionLog` | `"log"` | Log a message | | `ActionAbort` | `"abort"` | Abort workflow execution | | `ActionContinue` | `"continue"` | Continue despite error | | `ActionExport` | `"export"` | Export a variable | | `ActionRun` | `"run"` | Run a command or function | | `ActionNotify` | `"notify"` | Send notification | ## StepStatus The `StepStatus` type represents the execution state of a step. | Constant | Value | Description | | ------------------- | ----------- | --------------------------- | | `StepStatusPending` | `"pending"` | Step waiting to execute | | `StepStatusRunning` | `"running"` | Step currently executing | | `StepStatusSuccess` | `"success"` | Step completed successfully | | `StepStatusFailed` | `"failed"` | Step failed | | `StepStatusSkipped` | `"skipped"` | Step was skipped | ## RunStatus The `RunStatus` type represents the overall run state. | Constant | Value | Description | | -------------------- | ------------- | -------------------------- | | `RunStatusPending` | `"pending"` | Run waiting to start | | `RunStatusRunning` | `"running"` | Run in progress | | `RunStatusCompleted` | `"completed"` | Run completed successfully | | `RunStatusFailed` | `"failed"` | Run failed | | `RunStatusCancelled` | `"cancelled"` | Run was cancelled | | `RunStatusSkipped` | `"skipped"` | Run was skipped | ## Severity (Linter) The `Severity` type defines lint issue severity levels. | Constant | Value | Description | | ----------------- | ----- | ------------------- | | `SeverityInfo` | `0` | Informational issue | | `SeverityWarning` | `1` | Warning issue | | `SeverityError` | `2` | Error issue | ## OutputFormat (Linter) The `OutputFormat` type defines linter output formats. | Constant | Value | Description | | -------------- | ---------- | ------------------------------------ | | `FormatPretty` | `"pretty"` | Colored terminal output with context | | `FormatJSON` | `"json"` | Machine-readable JSON | | `FormatGitHub` | `"github"` | GitHub Actions annotations | ## OverrideMode (Inheritance) The `OverrideMode` type defines merge strategies for workflow inheritance. | Constant | Value | Description | | --------------------- | ----------- | ---------------------------------------------------- | | `OverrideModeReplace` | `"replace"` | Completely replace parent items with child items | | `OverrideModePrepend` | `"prepend"` | Add child items before parent items | | `OverrideModeAppend` | `"append"` | Add child items after parent items (default) | | `OverrideModeMerge` | `"merge"` | Match by name: replace, append new, remove specified | Used in the `override.steps.mode` and `override.modules.mode` fields when inheriting workflows. ## Summary Table | Category | Type | Values | | ------------- | -------------- | ---------------------------------------------------------------------------------------------- | | Workflow | `WorkflowKind` | `module`, `flow`, `fragment` | | Step | `StepType` | `bash`, `function`, `parallel-steps`, `foreach`, `remote-bash`, `http`, `llm`, `fragment-step` | | Trigger | `TriggerType` | `cron`, `event`, `watch`, `manual` | | Runner | `RunnerType` | `host`, `docker`, `ssh` | | Variable | `VariableType` | `domain`, `subdomain`, `url`, `cidr`, `path`, `file`, `folder`, `number`, `string`, `repo` | | Target | `TargetType` | `domain`, `subdomain`, `url`, `cidr`, `repo`, `path`, `file`, `folder`, `number`, `string` | | Action | `ActionType` | `log`, `abort`, `continue`, `export`, `run`, `notify` | | Status | `StepStatus` | `pending`, `running`, `success`, `failed`, `skipped` | | Run | `RunStatus` | `pending`, `running`, `completed`, `failed`, `cancelled`, `skipped` | | Lint Severity | `Severity` | `info`, `warning`, `error` | | Lint Format | `OutputFormat` | `pretty`, `json`, `github` | | Override | `OverrideMode` | `replace`, `prepend`, `append`, `merge` | # Default Variables Reference Source: https://docs.osmedeus.org/reference/variables Complete reference for template variables and utility functions in Osmedeus # Default Variables Reference Osmedeus workflows use template variables for dynamic content substitution. Variables are enclosed in double curly braces: `{{variable}}`. ## Template Syntax ### Standard Variables Standard variables use `{{variable}}` syntax and are evaluated at step execution time. ```yaml theme={null} command: "nuclei -l {{Output}}/urls.txt -t {{Data}}/templates/" ``` ### Secondary Variables (Foreach) Foreach loops use `[[variable]]` syntax to avoid conflicts with standard templates: ```yaml theme={null} - name: scan-each type: foreach input: "{{Output}}/subdomains.txt" variable: sub step: type: bash command: "httpx -u [[sub]] >> {{Output}}/httpx.txt" ``` ## Built-in Variables ### Path Variables | Variable | Description | Default | | -------------------------- | --------------------------------------------- | ------------------------------------------ | | `{{BaseFolder}}` | Base installation folder | `~/osmedeus-base` | | `{{base_folder}}` | Alias for BaseFolder | `~/osmedeus-base` | | `{{Binaries}}` | Path to binaries | `{{BaseFolder}}/external-binaries` | | `{{Data}}` | Path to data files | `{{BaseFolder}}/data` | | `{{ExternalData}}` | Alias for Data | `{{BaseFolder}}/data` | | `{{ExternalConfigs}}` | Path to external configs | `{{BaseFolder}}/configs` | | `{{ExternalScripts}}` | Path to external scripts | `{{BaseFolder}}/scripts` | | `{{ExternalAgentConfigs}}` | Path to agent configs | `{{BaseFolder}}/external-agent-configs` | | `{{ExternalAgents}}` | Alias for ExternalAgentConfigs | `{{BaseFolder}}/external-agent-configs` | | `{{Workflows}}` | Path to workflows | `{{BaseFolder}}/workflows` | | `{{MarkdownTemplates}}` | Path to markdown report templates | `{{BaseFolder}}/markdown-report-templates` | | `{{ExternalMarkdowns}}` | Alias for MarkdownTemplates | `{{BaseFolder}}/markdown-report-templates` | | `{{Workspaces}}` | Path to workspaces (overridable with -W flag) | `~/workspaces-osmedeus` | | `{{SnapshotsFolder}}` | Path to snapshots | `{{BaseFolder}}/snapshots` | ### Target Variables | Variable | Description | Example | | ----------------- | --------------------------------- | ---------------------------- | | `{{Target}}` | Current scan target | `example.com` | | `{{target}}` | Alias for Target | `example.com` | | `{{TargetFile}}` | File containing targets (-T flag) | `/path/to/targets.txt` | | `{{TargetSpace}}` | Sanitized target (workspace name) | `example_com` | | `{{Output}}` | Output directory for target | `{{Workspaces}}/example_com` | | `{{Workspace}}` | Alias for TargetSpace | `example_com` | | `{{workspace}}` | Alias for Workspace | `example_com` | ### Target Heuristics (Auto-Detected) These variables are automatically populated based on target type detection. Set `--heuristics none` to disable. | Variable | Description | Applies To | | --------------------- | ---------------- | ---------------------------------------- | | `{{TargetType}}` | Detected type | `url`, `domain`, `ip`, `file`, `unknown` | | `{{HeuristicsCheck}}` | Heuristics level | `basic` (default), `deep`, `none` | **URL Target Variables** (when `{{TargetType}}` is `url`): | Variable | Description | Example | | ------------------------- | ------------------------------ | -------------------------- | | `{{TargetBaseURL}}` | Base URL without path | `https://example.com:8080` | | `{{TargetRootURL}}` | Root URL (scheme+host) | `https://example.com` | | `{{TargetHostname}}` | Hostname from URL | `example.com` | | `{{TargetRootDomain}}` | Root domain | `example.com` | | `{{TargetTLD}}` | Top-level domain | `com` | | `{{TargetSLD}}` | Second-level domain | `example` | | `{{Org}}` | Alias for TargetSLD | `example` | | `{{TargetHost}}` | Host with port | `example.com:8080` | | `{{TargetPort}}` | Port number | `8080` | | `{{TargetPath}}` | URL path | `/api/v1` | | `{{TargetFileExt}}` | File extension | `html` | | `{{TargetScheme}}` | URL scheme | `https` | | `{{TargetStatusCode}}` | HTTP status code (if detected) | `200` | | `{{TargetContentLength}}` | Content length (if detected) | `1234` | **Domain Target Variables** (when `{{TargetType}}` is `domain`): | Variable | Description | Example | | ---------------------- | ---------------------------------- | --------------- | | `{{TargetRootDomain}}` | Root domain | `example.com` | | `{{TargetTLD}}` | Top-level domain | `com` | | `{{TargetSLD}}` | Second-level domain | `example` | | `{{Org}}` | Alias for TargetSLD | `example` | | `{{TargetIsWildcard}}` | Is wildcard subdomain | `true`, `false` | | `{{TargetResolvedIP}}` | Resolved IP address (if available) | `93.184.216.34` | **IP Target Variables** (when `{{TargetType}}` is `ip`): | Variable | Description | Example | | ---------------------- | ----------------- | ------------- | | `{{TargetRootDomain}}` | Original IP value | `192.168.1.1` | ### Platform Detection Variables | Variable | Description | Example Values | | --------------------------- | --------------------------- | ------------------------------ | | `{{PlatformOS}}` | Operating system | `linux`, `darwin`, `windows` | | `{{PlatformArch}}` | CPU architecture | `amd64`, `arm64` | | `{{PlatformInDocker}}` | Running in Docker container | `true`, `false` | | `{{PlatformInKubernetes}}` | Running in Kubernetes pod | `true`, `false` | | `{{PlatformCloudProvider}}` | Detected cloud provider | `aws`, `gcp`, `azure`, `local` | ### Thread Variables | Variable | Description | Default | | ----------------- | ------------------------------- | ------- | | `{{threads}}` | Thread count (tactic based) | `10` | | `{{baseThreads}}` | Base thread count (threads / 2) | `5` | ### Metadata Variables | Variable | Description | Example | | ------------------ | --------------------------------------------------- | -------------------------------------- | | `{{Version}}` | Osmedeus version | `5.0.0` | | `{{ModuleName}}` | Current workflow/module name | `subdomain-enum` | | `{{workflow}}` | Alias for ModuleName | `subdomain-enum` | | `{{FlowName}}` | Parent flow name (empty if running module directly) | `recon-flow` | | `{{RunUUID}}` | UUID for current run execution | `550e8400-e29b-41d4-a716-446655440000` | | `{{run_uuid}}` | Alias for RunUUID | `550e8400-e29b-41d4-a716-446655440000` | | `{{DBRunID}}` | Integer Run.ID for database foreign keys | `42` | | `{{TaskDate}}` | Task date | `2025-01-15` | | `{{Today}}` | Current date | `2025-01-15` | | `{{TimeStamp}}` | Unix timestamp | `1705312800` | | `{{CurrentTime}}` | Current time (ISO 8601) | `2025-01-15T10:00:00` | | `{{RandomString}}` | Random 6-char lowercase string | `xkmprq` | ### Constants | Variable | Description | Example | | --------------- | ------------------------- | ----------------- | | `{{DefaultUA}}` | Default User-Agent string | `Mozilla/5.0 ...` | ### State File Variables | Variable | Description | | ------------------------- | -------------------------- | | `{{StateExecutionLog}}` | Path to execution log | | `{{StateConsoleLog}}` | Path to console log | | `{{StateCompletedFile}}` | Path to run-completed.json | | `{{StateFile}}` | Path to run-state.json | | `{{StateWorkflowFile}}` | Path to run-workflow\.yaml | | `{{StateWorkflowFolder}}` | Path to run-modules folder | ### Chunk Mode Variables Used for distributed scanning with `--chunk` flag: | Variable | Description | | ----------------- | ------------------------- | | `{{ChunkIndex}}` | Current chunk index | | `{{ChunkSize}}` | Number of items per chunk | | `{{TotalChunks}}` | Total number of chunks | | `{{ChunkStart}}` | Start position | | `{{ChunkEnd}}` | End position | ### Event Trigger Variables These variables are only available for event-triggered workflows: | Variable | Description | Example | | -------------------- | ---------------------------------- | ---------------------------- | | `{{EventEnvelope}}` | Full event envelope as JSON string | `{"topic":"assets.new",...}` | | `{{EventTopic}}` | Event topic | `assets.new` | | `{{EventSource}}` | Event source | `nuclei` | | `{{EventDataType}}` | Event data type | `asset` | | `{{EventTimestamp}}` | Event timestamp | `2025-01-15T10:00:00Z` | | `{{EventData}}` | Event data payload as JSON string | `{"url":"https://...",...}` | ## Utility Functions Utility functions are executed via the Goja JavaScript runtime and can be used in function steps or template expressions. ### File Operations | Function | Returns | Description | | ------------------------------------------- | --------- | --------------------------------- | | `fileExists(path)` | bool | Check if file exists | | `fileLength(path)` | int | Count non-empty lines in file | | `dirLength(path)` | int | Count entries in directory | | `fileContains(path, pattern)` | bool | Check if file contains pattern | | `regexExtract(path, pattern)` | \[]string | Extract matching lines from file | | `readFile(path)` | string | Read entire file contents | | `readLines(path)` | \[]string | Read file as array of lines | | `removeFile(path)` | bool | Delete a file | | `removeFolder(path)` | bool | Delete folder recursively | | `rm_rf(path)` | bool | Delete file or folder recursively | | `remove_all_except(folder, keep)` | bool | Remove all except keep\_file | | `createFolder(path)` | bool | Create folder recursively | | `appendFile(dest, source)` | bool | Append source file to dest | | `moveFile(source, dest)` | bool | Move/rename file | | `glob(pattern)` | \[]string | List files matching glob pattern | | `grep_string(source, str)` | string | Return lines containing string | | `grep_regex(source, pattern)` | string | Return lines matching regex | | `grep_string_to_file(dest, source, str)` | bool | Write matching lines to file | | `grep_regex_to_file(dest, source, pattern)` | bool | Write matching lines to file | | `remove_blank_lines(path)` | bool | Remove blank lines in-place | ### String Operations | Function | Returns | Description | | ------------------------------------- | --------- | --------------------------------- | | `trim(str)` | string | Trim whitespace | | `split(str, delim)` | \[]string | Split by delimiter | | `join(arr, delim)` | string | Join with delimiter | | `replace(str, old, new)` | string | Replace all occurrences | | `contains(str, substr)` | bool | Check contains substring | | `startsWith(str, prefix)` | bool | Check starts with prefix | | `endsWith(str, suffix)` | bool | Check ends with suffix | | `toLowerCase(str)` | string | Convert to lowercase | | `toUpperCase(str)` | string | Convert to uppercase | | `match(str, pattern)` | bool | Check regex match | | `regex_match(pattern, str)` | bool | Check regex match (pattern first) | | `cut_with_delim(input, delim, field)` | string | Extract field (1-indexed) | | `normalize_path(input)` | string | Replace special chars with \_ | | `clean_sub(path, target?)` | bool | Clean and dedupe subdomains | ### Type Conversion | Function | Returns | Description | | ----------------- | ------- | ----------------------- | | `parseInt(str)` | int | Parse string to integer | | `parseFloat(str)` | float | Parse string to float | | `toString(val)` | string | Convert to string | | `toBoolean(val)` | bool | Convert to boolean | ### Utility | Function | Returns | Description | | ------------------- | ------- | -------------------------- | | `len(val)` | int | Get length of string/array | | `isEmpty(val)` | bool | Check if empty | | `isNotEmpty(val)` | bool | Check if not empty | | `printf(message)` | void | Print to stdout | | `cat_file(path)` | void | Print file content | | `exit(code)` | void | Exit with code | | `exec_cmd(command)` | string | Execute bash command | | `sleep(seconds)` | void | Pause execution | ### Logging | Function | Returns | Description | | -------------------- | ------- | ------------------------ | | `log_debug(message)` | void | Log with \[DEBUG] prefix | | `log_info(message)` | void | Log with \[INFO] prefix | | `log_warn(message)` | void | Log with \[WARN] prefix | | `log_error(message)` | void | Log with \[ERROR] prefix | ### HTTP | Function | Returns | Description | | ----------------------------------------- | ------- | ----------------- | | `httpRequest(url, method, headers, body)` | object | Make HTTP request | | `http_get(url)` | object | HTTP GET request | | `http_post(url, body)` | object | HTTP POST request | ### Generation | Function | Returns | Description | | ---------------------- | ------- | -------------------------- | | `randomString(length)` | string | Random alphanumeric string | | `uuid()` | string | Generate UUID v4 | ### Encoding | Function | Returns | Description | | ------------------- | ------- | ------------------ | | `base64Encode(str)` | string | Encode to base64 | | `base64Decode(str)` | string | Decode from base64 | ### Data Query | Function | Returns | Description | | --------------------------- | ------- | ---------------------------- | | `jq(jsonData, query)` | any | Extract data using jq syntax | | `jq_from_file(path, query)` | any | jq from JSON file | ### Notification | Function | Returns | Description | | ----------------------------------- | ------- | ---------------------- | | `notifyTelegram(message)` | bool | Send Telegram message | | `sendTelegramFile(path, caption?)` | bool | Send file to Telegram | | `notifyWebhook(message)` | bool | Send to all webhooks | | `sendWebhookEvent(eventType, data)` | bool | Send event to webhooks | ### CDN/Storage | Function | Returns | Description | | --------------------------------------------- | ------------ | --------------------------- | | `cdnUpload(localPath, remotePath)` | bool | Upload to cloud storage | | `cdnDownload(remotePath, localPath)` | bool | Download from cloud storage | | `cdnExists(remotePath)` | bool | Check if file exists | | `cdnDelete(remotePath)` | bool | Delete from cloud storage | | `cdnSyncUpload(localDir, remotePrefix)` | object | Sync directory to cloud | | `cdnSyncDownload(remotePrefix, localDir)` | object | Sync from cloud | | `cdnGetPresignedURL(remotePath, expiryMins?)` | string | Generate presigned URL | | `cdnList(prefix?)` | \[]object | List files with metadata | | `cdnStat(remotePath)` | object\|null | Get file metadata | ### Unix Commands | Function | Returns | Description | | --------------------------------------- | ------- | --------------------------- | | `sortUnix(input, output?)` | bool | Sort with LC\_ALL=C sort -u | | `wgetUnix(url, output?)` | bool | Download with wget | | `gitClone(repo, dest?)` | bool | Clone git repository | | `zipUnix(source, dest)` | bool | Create zip archive | | `unzipUnix(source, dest?)` | bool | Extract zip archive | | `tarUnix(source, dest)` | bool | Create tar.gz archive | | `untarUnix(source, dest?)` | bool | Extract tar.gz archive | | `diffUnix(file1, file2, output?)` | string | Compare files | | `sed_string_replace(syntax, src, dest)` | bool | String replacement | | `sed_regex_replace(syntax, src, dest)` | bool | Regex replacement | ### Archive (Go implementations) | Function | Returns | Description | | ------------------------- | ------- | -------------------------- | | `zip_dir(source, dest)` | bool | Zip using Go archive/zip | | `unzip_dir(source, dest)` | bool | Unzip using Go archive/zip | ### Diff | Function | Returns | Description | | --------------------------- | ------- | ------------------- | | `extractDiff(file1, file2)` | string | Lines only in file2 | ### Output | Function | Returns | Description | | ------------------------------------ | ------- | -------------------- | | `save_content(content, path)` | bool | Save content to file | | `jsonl_to_csv(source, dest)` | bool | Convert JSONL to CSV | | `csv_to_jsonl(source, dest)` | bool | Convert CSV to JSONL | | `jsonl_unique(source, dest, fields)` | bool | Deduplicate JSONL | | `jsonl_filter(source, dest, fields)` | bool | Filter JSONL fields | ### URL Processing | Function | Returns | Description | | ------------------------------------- | ------- | ---------------------- | | `interesting_urls(src, dest, field?)` | bool | Dedupe and filter URLs | ### Markdown | Function | Returns | Description | | ------------------------------------------ | ------- | ------------------------ | | `render_markdown_from_file(path)` | string | Render markdown | | `print_markdown_from_file(path)` | void | Print with highlighting | | `convert_jsonl_to_markdown(input, output)` | bool | JSONL to markdown table | | `convert_csv_to_markdown(path)` | string | CSV to markdown table | | `render_markdown_report(template, output)` | bool | Render report template | | `generate_security_report(template)` | bool | Generate security report | ### Database | Function | Returns | Description | | --------------------------------------------------------- | ------- | ----------------------------- | | `db_update(table, key, field, value)` | bool | Update database field | | `db_import_asset(workspace, json)` | bool | Import asset (upsert) | | `db_raw_insert_asset(workspace, json)` | int | Insert asset (returns ID) | | `db_import_asset_from_file(workspace, path)` | int | Import from JSONL | | `db_import_vuln(workspace, json)` | bool | Import vulnerability | | `db_import_vuln_from_file(workspace, path)` | int | Import vulns from JSONL | | `db_total_subdomains(path)` | int | Count and update workspace | | `db_total_urls(path)` | int | Count and update workspace | | `db_total_assets(path)` | int | Count and update workspace | | `db_total_vulns(path)` | int | Count and update workspace | | `db_vuln_critical(path)` | int | Count critical vulns | | `db_vuln_high(path)` | int | Count high vulns | | `db_vuln_medium(path)` | int | Count medium vulns | | `db_vuln_low(path)` | int | Count low vulns | | `db_total_ips(path)` | int | Count and update IPs | | `db_total_links(path)` | int | Count and update links | | `db_total_content(path)` | int | Count and update content | | `db_total_archive(path)` | int | Count and update archive | | `runtime_export()` | bool | Export run state | | `register_artifact(path, type?)` | bool | Register as artifact | | `store_artifact(path)` | bool | Store as artifact | | `db_select_assets(workspace, format)` | string | Select assets | | `db_select_assets_filtered(ws, status, type, fmt)` | string | Filtered assets | | `db_select_vulnerabilities(workspace, format)` | string | Select vulns | | `db_select_vulnerabilities_filtered(ws, sev, asset, fmt)` | string | Filtered vulns | | `db_select(sql_query, format)` | string | Execute SELECT | | `db_select_to_file(sql_query, dest)` | bool | SELECT to file | | `db_select_to_jsonl(sql_query, fields, dest)` | bool | SELECT to JSONL | | `db_select_total_subdomains()` | int | Get workspace subdomain count | | `db_select_total_urls()` | int | Get workspace URL count | | `db_select_total_assets()` | int | Get workspace asset count | | `db_select_total_vulns()` | int | Get workspace vuln count | | `db_select_vuln_critical()` | int | Get critical count | | `db_select_vuln_high()` | int | Get high count | | `db_select_vuln_medium()` | int | Get medium count | | `db_select_vuln_low()` | int | Get low count | | `db_asset_diff(workspace)` | string | Get asset diff JSONL | | `db_vuln_diff(workspace)` | string | Get vuln diff JSONL | | `db_asset_diff_to_file(workspace, dest)` | bool | Asset diff to file | | `db_vuln_diff_to_file(workspace, dest)` | bool | Vuln diff to file | ### Environment Functions | Function | Returns | Description | | ------------------------ | ------- | ------------------------------ | | `os_getenv(name)` | string | Get environment variable value | | `os_setenv(name, value)` | bool | Set environment variable | ### Installer Functions | Function | Returns | Description | | ----------------------------------------------- | ------- | ------------------------------------ | | `go_getter(url, dest)` | bool | Download files/repos using go-getter | | `go_getter_with_sshkey(ssh_key, git_url, dest)` | bool | Clone git repo with SSH key | | `nix_install(package, dest?)` | bool | Install package via Nix | ### LLM Functions | Function | Returns | Description | | --------------------------------------- | ------- | ----------------------------------------------------------------------- | | `llm_invoke(message)` | string | Simple LLM call with direct message | | `llm_invoke_custom(message, body_json)` | string | LLM call with custom POST body template (use `{{message}}` placeholder) | | `llm_conversations(msg1, msg2, ...)` | string | Multi-turn conversation with `role:content` format messages | ## Usage Examples ### Using Environment Functions ```yaml theme={null} - name: setup-api-key type: function functions: - "os_setenv('API_KEY', '{{api_token}}')" - "log_info('API key configured')" - name: read-config type: function function: "os_getenv('HOME')" exports: home_dir: "{{_result}}" ``` ### Using Installer Functions ```yaml theme={null} - name: install-tools type: function functions: - "go_getter('https://github.com/projectdiscovery/nuclei/releases/download/v3.0.0/nuclei_3.0.0_linux_amd64.zip', '{{Binaries}}')" - "nix_install('subfinder', '{{Binaries}}')" - name: clone-private-repo type: function function: "go_getter_with_sshkey('~/.ssh/id_rsa', 'git@github.com:user/private-templates.git', '{{Data}}/templates')" ``` ### Database Functions ```yaml theme={null} - name: update-stats type: function functions: - "db_total_subdomains('{{Output}}/subdomains.txt')" - "db_total_urls('{{Output}}/urls.txt')" - "db_import_asset_from_file('{{Workspace}}', '{{Output}}/httpx.jsonl')" - "db_import_vuln_from_file('{{Workspace}}', '{{Output}}/nuclei.jsonl')" ``` ### Conditional Logic with Functions ```yaml theme={null} - name: check-and-process type: function pre_condition: "fileExists('{{Output}}/results.txt')" function: "fileLength('{{Output}}/results.txt')" exports: result_count: "{{_result}}" - name: decide-next type: function function: "log_info('Results: ' + {{result_count}})" decision: switch: "{{result_count}}" cases: "0": { goto: _end } default: { goto: process-results } ``` # Workflow Schema Reference Source: https://docs.osmedeus.org/reference/workflow-schema Complete YAML schema reference for Osmedeus workflows # Workflow Schema Reference This document provides a complete reference for the Osmedeus workflow YAML schema. ## Top-Level Fields All workflows share these common top-level fields: ```yaml theme={null} kind: module | flow | fragment # Required: Workflow type name: workflow-name # Required: Unique identifier description: "Description text" # Optional: Human-readable description tags: "recon, fast" # Optional: Comma-separated tags extends: parent-workflow # Optional: Parent workflow to inherit from override: # Optional: Override sections from parent params: {} steps: {} modules: {} params: # Optional: Input parameters - name: param_name default: "value" required: false triggers: # Optional: Automated triggers - name: trigger-name on: cron | event | watch | manual enabled: true dependencies: # Optional: External tool dependencies binaries: - nuclei - httpx ``` ## Workflow Kinds | Kind | Purpose | Contains | | ---------- | ------------------------ | --------------- | | `module` | Single execution unit | `steps` array | | `flow` | Orchestrate modules | `modules` array | | `fragment` | Reusable step collection | `steps` array | ## Workflow Inheritance Workflows can extend parent workflows using the `extends` and `override` fields. ### Extends Field ```yaml theme={null} # By name (searched in workflow directory) extends: parent-workflow-name # By relative path extends: ./parent.yaml # By path from workflow directory extends: modules/base-module.yaml ``` ### Override Schema ```yaml theme={null} override: # Override specific parameter properties params: param_name: default: "new-value" type: "string" required: true generator: "generator_func" # Override steps (module workflows only) steps: mode: replace | prepend | append | merge steps: # Steps to add/replace - name: new-step type: bash command: "..." remove: # Step names to remove (merge mode only) - step-to-remove replace: # Steps to replace by name (merge mode only) - name: existing-step type: bash command: "new-command" # Override modules (flow workflows only) modules: mode: replace | prepend | append | merge modules: # Modules to add/replace - name: new-module path: modules/new.yaml remove: # Module names to remove (merge mode only) - module-to-remove replace: # Modules to replace by name (merge mode only) - name: existing-module path: modules/replacement.yaml # Replace triggers entirely triggers: - name: new-trigger on: cron schedule: "0 * * * *" # Merge with parent dependencies dependencies: commands: - new-tool files: - /path/to/file # Override preferences (child values override parent) preferences: disable_notifications: true silent: false # Override runner configuration runner_config: image: "custom-image:latest" env: NEW_VAR: "value" # Override runner type runner: docker ``` ### Override Modes | Mode | Description | | --------- | ------------------------------------------------------------- | | `replace` | Completely replace parent items with child items | | `prepend` | Add child items before parent items | | `append` | Add child items after parent items (default) | | `merge` | Match by name: replace matching, append new, remove specified | ### Inheritance Example ```yaml theme={null} # Parent: base-enum.yaml kind: module name: base-enum params: - name: threads default: 10 steps: - name: subfinder type: bash command: "subfinder -d {{Target}}" # Child: fast-enum.yaml kind: module name: fast-enum extends: base-enum override: params: threads: default: 50 steps: mode: append steps: - name: additional-tool type: bash command: "extra-tool -t {{Target}}" ``` ## Module-Specific Fields Modules contain steps that execute sequentially or based on dependencies. ```yaml theme={null} kind: module name: my-module # Runner configuration (optional) runner: host | docker | ssh runner_config: # Docker settings image: "ubuntu:latest" env: KEY: "value" volumes: - "/host/path:/container/path" network: "host" persistent: false # SSH settings host: "remote.example.com" port: 22 user: "username" key_file: "~/.ssh/id_rsa" workdir: "/tmp" # Fragment includes includes: - path: fragments/common-setup.yaml fragment_name: setup # Name for fragment-step reference params: custom_var: "{{Target}}" position: prepend | append # Steps array steps: - name: step-name type: bash command: "echo hello" ``` ## Flow-Specific Fields Flows orchestrate multiple modules with dependencies. ```yaml theme={null} kind: flow name: my-flow modules: - name: first-module path: modules/first.yaml params: threads: "10" - name: second-module path: modules/second.yaml depends_on: - first-module condition: "fileLength('{{Output}}/subdomains.txt') > 0" on_success: - action: log message: "Module completed" on_error: - action: notify notify: "Module failed" decision: switch: "{{status}}" cases: "critical": { goto: alert-module } "none": { goto: _end } default: { goto: next-module } ``` ## Fragment-Specific Fields Fragments are reusable step collections that can be included in modules. ```yaml theme={null} kind: fragment name: common-subdomain-enum params: - name: wordlist default: "{{Data}}/wordlists/subdomains.txt" steps: - name: subfinder type: bash command: "subfinder -d {{Target}} -o {{Output}}/subfinder.txt" - name: amass type: bash command: "amass enum -d {{Target}} -o {{Output}}/amass.txt" ``` ## Step Types ### Step Type: bash Execute shell commands locally. ```yaml theme={null} - name: run-nuclei type: bash command: "nuclei -l {{Output}}/urls.txt -o {{Output}}/nuclei.json" timeout: 1h # Multiple sequential commands - name: multi-commands type: bash commands: - "echo 'Step 1'" - "echo 'Step 2'" # Parallel commands - name: parallel-scans type: bash parallel_commands: - "nuclei -l urls.txt -t cves/" - "nuclei -l urls.txt -t exposed-panels/" ``` ### Step Type: function Execute utility functions via the Goja JavaScript runtime. ```yaml theme={null} - name: check-results type: function function: "fileExists('{{Output}}/results.txt')" exports: has_results: "{{_result}}" # Multiple functions - name: process-data type: function functions: - "log_info('Processing started')" - "sortUnix('{{Output}}/urls.txt')" - "remove_blank_lines('{{Output}}/urls.txt')" ``` ### Step Type: parallel-steps Execute multiple steps concurrently. ```yaml theme={null} - name: parallel-enum type: parallel-steps parallel_steps: - name: subfinder type: bash command: "subfinder -d {{Target}} -o {{Output}}/subfinder.txt" - name: amass type: bash command: "amass enum -d {{Target}} -o {{Output}}/amass.txt" ``` ### Step Type: foreach Iterate over input lines with parallel processing. ```yaml theme={null} - name: scan-subdomains type: foreach input: "{{Output}}/subdomains.txt" variable: subdomain threads: 10 step: name: httpx-probe type: bash command: "httpx -u [[subdomain]] >> {{Output}}/httpx.txt" ``` ### Step Type: remote-bash Execute commands in Docker containers or via SSH. ```yaml theme={null} # Docker execution - name: docker-scan type: remote-bash step_runner: docker step_runner_config: image: "projectdiscovery/nuclei:latest" volumes: - "{{Output}}:/output" command: "nuclei -l /output/urls.txt -o /output/nuclei.json" step_remote_file: "/output/nuclei.json" host_output_file: "{{Output}}/nuclei.json" # SSH execution - name: ssh-scan type: remote-bash step_runner: ssh step_runner_config: host: "scan-server.example.com" user: "scanner" key_file: "~/.ssh/id_rsa" command: "nuclei -l /tmp/urls.txt" ``` ### Step Type: http Make HTTP requests and capture responses. ```yaml theme={null} - name: api-call type: http url: "https://api.example.com/scan" method: POST headers: Authorization: "Bearer {{api_token}}" Content-Type: "application/json" request_body: '{"target": "{{Target}}"}' exports: response: "{{_result}}" ``` ### Step Type: llm Interact with LLM APIs (OpenAI-compatible). ```yaml theme={null} - name: analyze-vulns type: llm messages: - role: system content: "You are a security analyst." - role: user content: "Analyze these findings: {{findings}}" llm_config: model: "gpt-4" temperature: 0.7 max_tokens: 2000 exports: analysis: "{{_result}}" # Embeddings - name: embed-text type: llm is_embedding: true embedding_input: - "text to embed" exports: embedding: "{{_result}}" ``` ### Step Type: fragment-step Execute a fragment inline with optional overrides. ```yaml theme={null} # Include fragment in module includes: - path: fragments/subdomain-enum.yaml fragment_name: subdomain-enum steps: - name: run-subdomain-enum type: fragment-step fragment_name: subdomain-enum override: threads: "20" # Override step field wordlist: "{{custom_wordlist}}" # Override template variable ``` ## Common Step Fields These fields are available on all step types: ```yaml theme={null} - name: step-name # Required: Unique step identifier type: bash # Required: Step type depends_on: # Optional: Step dependencies (DAG) - previous-step pre_condition: "fileExists('{{file}}')" # Optional: Skip if false timeout: 30m # Optional: Step timeout (e.g., 30, 30s, 30m, 1h, 1d) log: "{{Output}}/step.log" # Optional: Log file path exports: # Optional: Export variables result_count: "{{_result}}" on_success: # Optional: Success handlers - action: log message: "Step completed" on_error: # Optional: Error handlers - action: abort message: "Critical failure" decision: # Optional: Conditional routing switch: "{{status}}" cases: "critical": { goto: alert-step } default: { goto: next-step } ``` ## Decision Routing Steps support conditional branching using switch/case syntax: ```yaml theme={null} decision: switch: "{{variable}}" cases: "value1": { goto: step-a } "value2": { goto: step-b } default: { goto: fallback } ``` Use `goto: _end` to terminate the workflow. ## Parameters Parameters define workflow inputs with validation: ```yaml theme={null} params: - name: threads type: number default: 10 required: false description: "Number of threads" - name: wordlist type: file default: "{{Data}}/wordlists/common.txt" required: false - name: target_type type: string required: true ``` ## Triggers Triggers define automated execution: ```yaml theme={null} triggers: # Cron trigger - name: daily-scan on: cron schedule: "0 2 * * *" enabled: true # Event trigger - name: on-new-asset on: event topic: "assets.new" filter: asset_type: "subdomain" enabled: true # File watch trigger - name: watch-targets on: watch paths: - "{{Data}}/targets.txt" enabled: true # Manual (default) - name: manual on: manual enabled: true ``` ## Complete Examples ### Module Example ```yaml theme={null} kind: module name: subdomain-enum description: "Enumerate subdomains for a target domain" tags: "recon, subdomain" params: - name: threads default: 10 - name: wordlist default: "{{Data}}/wordlists/subdomains.txt" steps: - name: subfinder type: bash command: "subfinder -d {{Target}} -all -o {{Output}}/subfinder.txt" timeout: 30m - name: merge-results type: function depends_on: - subfinder functions: - "sortUnix('{{Output}}/subfinder.txt', '{{Output}}/subdomains.txt')" - "db_total_subdomains('{{Output}}/subdomains.txt')" ``` ### Flow Example ```yaml theme={null} kind: flow name: full-recon description: "Complete reconnaissance workflow" tags: "recon, full" modules: - name: subdomain-enum path: modules/subdomain-enum.yaml params: threads: "20" - name: port-scan path: modules/port-scan.yaml depends_on: - subdomain-enum condition: "fileLength('{{Output}}/subdomains.txt') > 0" - name: vuln-scan path: modules/vuln-scan.yaml depends_on: - port-scan ``` ### Fragment Example ```yaml theme={null} kind: fragment name: common-cleanup description: "Common cleanup steps" steps: - name: deduplicate type: function function: "sortUnix('{{input_file}}')" - name: remove-blanks type: function function: "remove_blank_lines('{{input_file}}')" ``` ### Module Using Fragment ```yaml theme={null} kind: module name: subdomain-scan includes: - path: fragments/common-cleanup.yaml fragment_name: cleanup steps: - name: subfinder type: bash command: "subfinder -d {{Target}} -o {{Output}}/subs-raw.txt" - name: cleanup-results type: fragment-step fragment_name: cleanup override: input_file: "{{Output}}/subs-raw.txt" ``` # Control Flow Source: https://docs.osmedeus.org/workflows/control-flow Conditions, routing, and error handling Control execution with conditions, handlers, and decision routing. ## Pre-Conditions Skip a step if a condition is false. ```yaml theme={null} - name: nuclei-scan type: bash pre_condition: 'fileLength("{{Output}}/live.txt") > 0' command: nuclei -l {{Output}}/live.txt -o {{Output}}/vulns.txt ``` ### Common Conditions ```yaml theme={null} # File exists pre_condition: 'fileExists("{{Output}}/targets.txt")' # File has content pre_condition: 'fileLength("{{Output}}/hosts.txt") > 0' # Check export value pre_condition: '{{host_count}} > 10' # Check parameter pre_condition: '{{enable_scan}} == "true"' # Combine conditions pre_condition: 'fileExists("{{Output}}/subs.txt") && {{threads}} > 0' ``` ### Condition Functions | Function | Description | | ----------------------- | --------------------------------- | | `fileExists(path)` | True if file exists | | `fileLength(path)` | Number of non-empty lines | | `dirLength(path)` | Number of directory entries | | `isEmpty(str)` | True if string is empty | | `contains(str, substr)` | True if string contains substring | ## Step Dependencies (DAG Execution) Steps can declare dependencies on other steps using the `depends_on` field. This enables: * Parallel execution of independent steps * Automatic ordering based on dependencies * DAG (Directed Acyclic Graph) execution ### Basic Dependencies ```yaml theme={null} steps: - name: subfinder type: bash command: "subfinder -d {{Target}} -o {{Output}}/subfinder.txt" - name: amass type: bash command: "amass enum -d {{Target}} -o {{Output}}/amass.txt" - name: merge-results type: function depends_on: - subfinder - amass functions: - "appendFile('{{Output}}/all-subs.txt', '{{Output}}/subfinder.txt')" - "appendFile('{{Output}}/all-subs.txt', '{{Output}}/amass.txt')" ``` In this example: * `subfinder` and `amass` run in parallel (no dependencies) * `merge-results` waits for both to complete before executing ### DAG Execution The executor builds a dependency graph and uses topological sorting (Kahn's algorithm) to determine execution order: ``` ┌───────────┐ ┌───────────┐ │ subfinder │ │ amass │ └─────┬─────┘ └─────┬─────┘ │ │ │ ┌──────────────┘ │ │ ▼ ▼ ┌─────────────┐ │merge-results│ └──────┬──────┘ │ ▼ ┌─────────────┐ │ http-probe │ └─────────────┘ ``` Steps at the same "level" (no dependencies between them) execute concurrently. ### Multiple Dependencies ```yaml theme={null} - name: final-report type: bash depends_on: - vuln-scan - port-scan - screenshot command: "generate-report {{Output}}" ``` The step waits for all listed dependencies to complete successfully. ### Cascade Failure If a dependency fails: 1. The dependent step is marked as failed (not executed) 2. All downstream steps are also marked as failed 3. Independent branches continue execution ```yaml theme={null} # If subfinder fails: # - merge-results is skipped # - amass continues (independent) ``` ### Dependencies vs Sequential Execution Without `depends_on`, steps execute sequentially in order: ```yaml theme={null} steps: - name: step1 # Runs first type: bash command: "..." - name: step2 # Runs second (waits for step1) type: bash command: "..." ``` With `depends_on`, steps can run in parallel: ```yaml theme={null} steps: - name: step1 # Runs first (parallel with step2) type: bash command: "..." - name: step2 # Runs first (parallel with step1) type: bash command: "..." - name: step3 # Waits for both type: bash depends_on: [step1, step2] command: "..." ``` ### Linter Validation The workflow linter validates dependencies: | Rule | Description | | --------------------- | --------------------------------------- | | `invalid-depends-on` | Dependency references non-existent step | | `circular-dependency` | Circular dependencies detected | ```bash theme={null} osmedeus workflow lint my-workflow.yaml ``` ### Flow Module Dependencies Flows also support `depends_on` for modules: ```yaml theme={null} kind: flow name: full-pipeline modules: - name: subdomain-enum path: modules/subdomain-enum.yaml - name: port-scan path: modules/port-scan.yaml - name: http-probe path: modules/http-probe.yaml depends_on: - subdomain-enum - port-scan ``` ## Decision Routing Route to different steps based on variable values using switch/case syntax. ```yaml theme={null} steps: - name: check-hosts type: bash command: wc -l < {{Output}}/hosts.txt exports: count: "{{stdout}}" decision: switch: "{{count}}" cases: "0": { goto: no-hosts-found } default: { goto: continue-scan } - name: no-hosts-found type: function function: log_warning("No hosts found, skipping scan") decision: switch: "true" cases: "true": { goto: _end } # Special: end workflow - name: continue-scan type: bash command: nuclei -l {{Output}}/hosts.txt -t {{Data}}/templates/ ``` ### Decision Syntax ```yaml theme={null} decision: switch: "{{variable}}" # Template expression to evaluate cases: # Map of values to actions "value1": { goto: step-a } "value2": { goto: step-b } default: { goto: fallback } # Optional: when no case matches ``` * `switch`: Template variable evaluated at runtime (exact string match) * `cases`: Map of string values to goto targets * `default`: Fallback when no case matches (optional) * `goto`: Target step name or `_end` to terminate ### Inline Execution in Cases Cases support running commands or functions inline, not just `goto`: ```yaml theme={null} decision: switch: "{{target_type}}" cases: "domain": goto: subdomain-enum command: "echo 'Processing domain target'" "ip": goto: port-scan commands: - "echo 'Processing IP target'" - "mkdir -p {{Output}}/ip-results" "url": function: "log_info('Direct URL target detected')" goto: web-scan ``` Each `DecisionCase` supports: | Field | Description | | ----------- | ----------------------------- | | `goto` | Target step name or `_end` | | `command` | Single command to execute | | `commands` | Multiple commands to execute | | `function` | Single function to execute | | `functions` | Multiple functions to execute | ### Condition-Based Decision Routing In addition to switch/case, decisions support `conditions` — an array of JavaScript expressions evaluated at runtime. All matching conditions execute (no short-circuit): ```yaml theme={null} - name: smart-routing type: function function: log_info("Evaluating conditions") decision: conditions: - if: "file_length('{{Output}}/subdomains.txt') > 0" goto: process-subdomains - if: "file_length('{{Output}}/subdomains.txt') > 100" command: "echo 'Large target set detected'" - if: "{{enableNmap}} == 'true' && contains('{{Port}}', '-')" commands: - "nmap -p {{Port}} {{Target}} -oN {{Output}}/nmap.txt" - "echo 'Port range scan complete'" - if: "{{scan_mode}} == 'aggressive'" functions: - "log_warning('Aggressive mode enabled')" - "log_info('Increasing thread count')" ``` #### Condition Fields | Field | Required | Description | | ----------- | -------- | ----------------------------------------------------- | | `if` | Yes | JavaScript expression (must evaluate to truthy/falsy) | | `goto` | No | Target step name or `_end` | | `command` | No | Single command to execute if condition is true | | `commands` | No | Multiple commands to execute if condition is true | | `function` | No | Single function to execute if condition is true | | `functions` | No | Multiple functions to execute if condition is true | All matching conditions in the array are executed — there is no short-circuit behavior. If you need exclusive routing, use switch/case instead. ### Special Goto Targets | Target | Description | | ----------- | ------------------------ | | `_end` | End workflow immediately | | `step-name` | Jump to named step | ## Success Handlers Execute actions when a step succeeds. ```yaml theme={null} - name: scan type: bash command: nuclei -l {{Output}}/hosts.txt -o {{Output}}/vulns.txt on_success: - action: log message: "Scan completed successfully" - action: export key: scan_status value: "completed" - action: notify message: "Vulnerability scan finished for {{target}}" ``` ### Available Actions | Action | Description | Parameters | | ---------- | ------------------ | -------------- | | `log` | Log a message | `message` | | `export` | Export a value | `key`, `value` | | `run` | Run a command | `command` | | `notify` | Send notification | `message` | | `continue` | Continue execution | - | ```yaml theme={null} on_success: - action: log message: "Step completed" - action: export key: result value: "success" - action: run command: echo "Done" >> {{Output}}/log.txt - action: notify message: "{{target}} scan finished" ``` ## Error Handlers Handle step failures. ```yaml theme={null} - name: risky-scan type: bash command: aggressive-tool {{target}} on_error: - action: log message: "Scan failed, continuing with fallback" - action: continue # Don't stop workflow - action: run command: fallback-tool {{target}} ``` ### Error Action Types | Action | Description | | ---------- | ----------------------- | | `log` | Log error message | | `abort` | Stop workflow (default) | | `continue` | Continue to next step | | `run` | Run recovery command | | `notify` | Send error notification | ```yaml theme={null} on_error: - action: abort # Stop workflow on error # OR on_error: - action: continue # Ignore error, continue ``` ## Combined Example ```yaml theme={null} steps: - name: enumerate type: bash command: subfinder -d {{target}} -o {{Output}}/subs.txt exports: sub_count: "{{stdout}}" on_success: - action: log message: "Found subdomains" on_error: - action: log message: "Enumeration failed" - action: continue - name: validate-results type: function function: fileLength("{{Output}}/subs.txt") exports: count: "{{result}}" decision: switch: "{{count}}" cases: "0": { goto: no-results } default: { goto: probe-hosts } - name: no-results type: function function: log_warning("No subdomains found for {{target}}") decision: switch: "true" cases: "true": { goto: _end } - name: probe-hosts type: bash pre_condition: 'fileLength("{{Output}}/subs.txt") > 0' command: httpx -l {{Output}}/subs.txt -o {{Output}}/live.txt on_success: - action: export key: probe_status value: "done" - action: notify message: "Probing complete for {{target}}" on_error: - action: log message: "HTTP probing failed" - action: abort - name: screenshot type: bash pre_condition: 'fileLength("{{Output}}/live.txt") > 0 && {{probe_status}} == "done"' command: gowitness file -f {{Output}}/live.txt -P {{Output}}/screenshots ``` ## Flow-Level Conditions Conditional module execution in flows: ```yaml theme={null} kind: flow name: conditional-flow params: - name: target - name: enable_active default: "false" modules: - name: passive-recon path: modules/passive.yaml - name: active-scan path: modules/active.yaml depends_on: [passive-recon] condition: '{{enable_active}} == "true"' - name: vuln-scan path: modules/vuln.yaml depends_on: [passive-recon] condition: 'fileLength("{{Output}}/live.txt") > 0' ``` ## Branching Patterns ### If-Then-Else ```yaml theme={null} - name: check type: function function: fileLength("{{Output}}/data.txt") exports: has_data: "{{result}}" decision: switch: "{{has_data}}" cases: "0": { goto: handle-empty } default: { goto: process-data } - name: process-data type: bash command: process {{Output}}/data.txt decision: switch: "true" cases: "true": { goto: finalize } - name: handle-empty type: function function: log_warning("No data to process") decision: switch: "true" cases: "true": { goto: finalize } - name: finalize type: function function: log_info("Workflow complete") ``` ### Early Exit ```yaml theme={null} - name: validate type: function function: fileExists("{{Output}}/required.txt") exports: valid: "{{result}}" decision: switch: "{{valid}}" cases: "false": { goto: _end } # Exit if invalid default: { goto: continue-scan } - name: continue-scan type: bash command: scan {{target}} ``` ### Loop with Retry ```yaml theme={null} - name: attempt-scan type: bash command: flaky-scanner {{target}} exports: attempt: "1" failed: "false" on_error: - action: export key: failed value: "true" - action: continue - name: retry-check type: function function: log_info("Checking retry status") decision: switch: "{{failed}}" cases: "false": { goto: success } "true": { goto: check-attempts } default: { goto: success } - name: check-attempts type: function function: log_info("Attempt {{attempt}}") decision: switch: "{{attempt}}" cases: "3": { goto: give-up } default: { goto: retry-scan } - name: retry-scan type: bash command: flaky-scanner {{target}} --retry exports: attempt: "{{parseInt({{attempt}}) + 1}}" on_error: - action: continue decision: switch: "true" cases: "true": { goto: retry-check } ``` ## Best Practices 1. **Always check file existence before processing** ```yaml theme={null} pre_condition: 'fileExists("{{Output}}/input.txt")' ``` 2. **Use meaningful log messages** ```yaml theme={null} on_success: - action: log message: "Found {{count}} subdomains for {{target}}" ``` 3. **Handle errors gracefully** ```yaml theme={null} on_error: - action: log message: "Step failed, attempting fallback" - action: continue ``` 4. **Use decision routing for complex logic** ```yaml theme={null} decision: switch: "{{dataset_size}}" cases: "large": { goto: large-dataset-handler } default: { goto: standard-handler } ``` 5. **End workflows cleanly** ```yaml theme={null} decision: switch: "{{fatal_error}}" cases: "true": { goto: _end } ``` ## Next Steps * [Step Types](step-types) - All step types * [Variables](variables) - Exports and conditions * [Functions Reference](../functions/reference) - Condition functions # Flow Workflows Source: https://docs.osmedeus.org/workflows/flows Multi-module orchestration with dependencies Flows orchestrate multiple modules with dependencies and conditional execution. ## Basic Flow ```yaml theme={null} kind: flow name: basic-recon description: Basic reconnaissance flow params: - name: target required: true modules: - name: subdomain-enum path: modules/subdomain-enum.yaml - name: http-probe path: modules/http-probe.yaml depends_on: - subdomain-enum - name: screenshot path: modules/screenshot.yaml depends_on: - http-probe ``` ## Module Reference Fields ```yaml theme={null} modules: - name: module-name # Required: reference name path: path/to/module.yaml # Required: module file path depends_on: # Optional: dependency list - other-module condition: 'expression' # Optional: skip condition params: # Optional: parameter overrides key: value ``` ### name Unique identifier for this module reference within the flow. ```yaml theme={null} - name: subdomain-enum # Used in depends_on references ``` ### path Path to the module YAML file (relative to workflow folder). ```yaml theme={null} - name: nuclei path: modules/nuclei-scan.yaml ``` ### depends\_on List of module names that must complete before this module runs. ```yaml theme={null} - name: vuln-scan path: modules/vuln.yaml depends_on: - subdomain-enum - http-probe # Both must complete first ``` ### condition JavaScript expression evaluated before module execution. Module is skipped if false. ```yaml theme={null} - name: screenshot path: modules/screenshot.yaml depends_on: [http-probe] condition: 'fileLength("{{Output}}/live-hosts.txt") > 0' ``` Common conditions: ```yaml theme={null} # File has content condition: 'fileLength("{{Output}}/data.txt") > 0' # File exists condition: 'fileExists("{{Output}}/targets.txt")' # Parameter check condition: '{{enable_vuln_scan}} == "true"' ``` ### params Override module parameters. ```yaml theme={null} - name: nuclei-fast path: modules/nuclei-scan.yaml params: threads: "100" severity: "critical,high" ``` ## Inline Modules Modules can define steps directly instead of referencing an external YAML file. This is useful for simple, self-contained tasks that don't warrant a separate file. ```yaml theme={null} kind: flow name: quick-recon modules: - name: quick-check description: Inline health check steps: - name: ping type: bash command: ping -c 1 {{Target}} - name: http-check type: bash command: curl -s -o /dev/null -w '%{http_code}' https://{{Target}} exports: status_code: "{{stdout}}" - name: full-scan path: modules/full-scan.yaml depends_on: [quick-check] condition: '{{status_code}} == "200"' ``` ### Inline Module Fields | Field | Description | | --------------- | ------------------------------------------------------- | | `name` | Required. Module reference name | | `steps` | Inline steps (makes this an inline module) | | `description` | Description for the inline module | | `runner` | Runner type for inline module (`host`, `docker`, `ssh`) | | `runner_config` | Runner configuration for inline module | When `steps` is defined, the module is treated as inline — no `path` is needed. Inline modules support the same step types and features as external module files. ### Inline Module with Runner ```yaml theme={null} modules: - name: docker-scan description: Run nuclei in Docker runner: docker runner_config: image: projectdiscovery/nuclei:latest volumes: - "{{Output}}:/output" steps: - name: nuclei type: bash command: nuclei -u {{Target}} -o /output/nuclei.txt ``` ## Dependency Graph Flows create a directed acyclic graph (DAG): ```yaml theme={null} modules: - name: A path: modules/a.yaml - name: B path: modules/b.yaml depends_on: [A] - name: C path: modules/c.yaml depends_on: [A] - name: D path: modules/d.yaml depends_on: [B, C] ``` Execution: ``` A # Step 1: A runs / \ B C # Step 2: B and C run in parallel \ / D # Step 3: D runs after B and C ``` ## Complex Flow Example ```yaml theme={null} kind: flow name: full-assessment description: Complete security assessment params: - name: target required: true - name: enable_active default: "false" description: Enable active scanning modules: # Passive reconnaissance - name: subdomain-enum path: modules/subdomain-enum.yaml - name: dns-enum path: modules/dns-enum.yaml # Both can run in parallel, both depend on nothing # Execution: subdomain-enum and dns-enum run together - name: http-probe path: modules/http-probe.yaml depends_on: - subdomain-enum - dns-enum # Waits for both to complete - name: screenshot path: modules/screenshot.yaml depends_on: [http-probe] condition: 'fileLength("{{Output}}/live.txt") > 10' - name: content-discovery path: modules/content-discovery.yaml depends_on: [http-probe] # Active scanning (conditional) - name: port-scan path: modules/port-scan.yaml depends_on: [subdomain-enum] condition: '{{enable_active}} == "true"' - name: vuln-scan path: modules/vuln-scan.yaml depends_on: - http-probe - port-scan condition: '{{enable_active}} == "true" && fileExists("{{Output}}/live.txt")' # Reporting - name: generate-report path: modules/report.yaml depends_on: - screenshot - content-discovery - vuln-scan ``` ## Running Flows ```bash theme={null} # Basic execution osmedeus run -f full-assessment -t example.com # With parameters osmedeus run -f full-assessment -t example.com -p 'enable_active=true' # Exclude specific modules osmedeus run -f full-assessment -t example.com -x vuln-scan -x port-scan # Dry run (preview) osmedeus run -f full-assessment -t example.com --dry-run ``` ## Module Exclusion Skip specific modules using exact match (`-x`) or substring match (`-X`): ```bash theme={null} # Exact match - exclude specific modules by name osmedeus run -f full-assessment -t target -x screenshot -x port-scan # Fuzzy match - exclude modules whose name contains the substring osmedeus run -f full-assessment -t target -X nmap -X nuclei ``` | Flag | Description | | ------------------------------ | ---------------------------------------------------------- | | `-x, --exclude ` | Exclude module by exact name (repeatable) | | `-X, --fuzzy-exclude ` | Exclude modules whose name contains substring (repeatable) | Excluded modules are treated as completed (dependencies are satisfied). ## Flow-Level vs Module-Level Params ```yaml theme={null} # Flow definition kind: flow name: my-flow params: - name: target required: true - name: threads default: "50" # Flow-level default modules: - name: scan path: modules/scan.yaml params: threads: "{{threads}}" # Use flow param wordlist: "/custom.txt" # Module-specific override ``` Resolution order: 1. Module reference `params` (highest priority) 2. Flow-level `params` 3. Module's own default params (lowest priority) ## Error Handling By default, if a module fails: * Dependent modules are skipped * Other independent branches continue ```yaml theme={null} modules: - name: A path: modules/a.yaml - name: B path: modules/b.yaml depends_on: [A] # If A fails, B is skipped - name: C path: modules/c.yaml # C runs regardless of A or B ``` ## Best Practices 1. **Group related modules** ```yaml theme={null} # Passive recon group - name: subdomain-enum - name: dns-enum # Active scan group - name: port-scan - name: vuln-scan ``` 2. **Use conditions for optional modules** ```yaml theme={null} - name: active-scan condition: '{{enable_active}} == "true"' ``` 3. **Check file existence before processing** ```yaml theme={null} - name: process-results condition: 'fileLength("{{Output}}/data.txt") > 0' ``` 4. **Parameterize module behavior** ```yaml theme={null} - name: nuclei params: threads: "{{threads}}" severity: "{{severity}}" ``` 5. **Add a final reporting module** ```yaml theme={null} - name: report depends_on: [all, other, modules] ``` ## Next Steps * [Variables](variables) - Parameter propagation * [Control Flow](control-flow) - Conditions in detail * [Step Types](step-types) - Module step types # Workflow Overview Source: https://docs.osmedeus.org/workflows/overview Introduction to workflow structure and concepts Workflows are YAML files that define automated scanning pipelines. ## Basic Structure ### Module ```yaml theme={null} kind: module # Type: module or flow name: my-workflow # Unique workflow name description: What it does # Human-readable description tags: # Optional categorization - reconnaissance - subdomain params: # Input parameters - name: target required: true runner: host # Execution environment runner_config: {} # Runner options triggers: # Scheduling options - name: daily on: cron schedule: "0 2 * * *" steps: # Execution steps - name: step-one type: bash command: echo "Hello" ``` ### Flow ```yaml theme={null} kind: flow name: my-pipeline description: Multi-module pipeline params: - name: target required: true modules: # Module references - name: recon path: modules/recon.yaml - name: scan path: modules/scan.yaml depends_on: [recon] ``` ## Field Reference ### Top-Level Fields | Field | Required | Description | | --------------- | -------- | --------------------------------------------- | | `kind` | Yes | `module` or `flow` | | `name` | Yes | Unique workflow identifier | | `description` | No | Human-readable description | | `tags` | No | Array of category tags | | `params` | No | Input parameter definitions | | `runner` | No | Default runner type (`host`, `docker`, `ssh`) | | `runner_config` | No | Runner configuration object | | `trigger` | No | Scheduling trigger definitions | | `steps` | Module | List of execution steps | | `modules` | Flow | List of module references | ### Parameters ```yaml theme={null} params: - name: target # Parameter name (required) required: true # Must be provided (default: false) default: "" # Default value description: Target domain ``` Use in templates as `{{target}}`. ### Tags ```yaml theme={null} tags: - reconnaissance - subdomain - passive ``` Filter workflows: ```bash theme={null} osmedeus workflow list --tags reconnaissance ``` ## Workflow Kinds ### Module Workflows For single, focused tasks: ```yaml theme={null} kind: module name: subdomain-enum params: - name: target required: true steps: - name: subfinder type: bash command: subfinder -d {{target}} -o {{Output}}/subs.txt - name: amass type: bash command: amass enum -passive -d {{target}} >> {{Output}}/subs.txt - name: dedupe type: bash command: sort -u {{Output}}/subs.txt -o {{Output}}/subdomains.txt ``` ### Flow Workflows For multi-stage pipelines: ```yaml theme={null} kind: flow name: full-recon params: - name: target required: true modules: - name: subdomain-enum path: modules/subdomain-enum.yaml - name: http-probe path: modules/http-probe.yaml depends_on: [subdomain-enum] - name: screenshot path: modules/screenshot.yaml depends_on: [http-probe] condition: 'fileLength("{{Output}}/live.txt") > 0' ``` ## Module References (Flows) ```yaml theme={null} modules: - name: module-name # Reference name path: modules/file.yaml # Path to module YAML depends_on: [dep1, dep2] # Wait for these modules condition: 'expression' # Skip if false params: # Override parameters key: value ``` ## Workflow Location Store workflows in the workflow folder (default: `~/osmedeus-base/workflows/`): ``` workflows/ ├── modules/ │ ├── subdomain-enum.yaml │ ├── http-probe.yaml │ └── nuclei-scan.yaml └── flows/ ├── basic-recon.yaml └── full-assessment.yaml ``` ## Running Workflows ```bash theme={null} # Run a module osmedeus run -m subdomain-enum -t example.com # Run a flow osmedeus run -f full-recon -t example.com # List available workflows osmedeus workflow list # Show workflow details osmedeus workflow show subdomain-enum # Validate workflow osmedeus workflow validate subdomain-enum ``` ## Validation and Linting ### Parser Validation The parser validates: 1. **Required fields**: `kind`, `name`, `steps` (module) or `modules` (flow) 2. **Valid kind**: Must be `module` or `flow` 3. **Step names**: Each step must have a unique name 4. **Step types**: Must be valid (bash, function, foreach, parallel-steps, remote-bash, http, llm, agent) 5. **Module paths**: Referenced modules must exist (flows) 6. **Circular dependencies**: No cycles in dependency graph (flows) ### Workflow Linter The workflow linter provides additional best-practice checks beyond basic parsing. It helps catch potential issues before runtime while allowing workflows to execute even with warnings. ```bash theme={null} # Lint workflows osmedeus workflow lint my-workflow.yaml osmedeus workflow lint my-workflow # By name osmedeus workflow lint /path/to/workflows/ # Directory # Output formats osmedeus workflow lint workflow.yaml --format pretty # Default osmedeus workflow lint workflow.yaml --format json # Machine-readable osmedeus workflow lint workflow.yaml --format github # CI annotations # Filter by severity osmedeus workflow lint workflow.yaml --severity info # All issues (default) osmedeus workflow lint workflow.yaml --severity warning # Warnings and errors osmedeus workflow lint workflow.yaml --severity error # Errors only # Disable specific rules osmedeus workflow lint workflow.yaml --disable unused-variable # CI mode (exit code 1 if errors) osmedeus workflow lint workflow.yaml --check ``` ### Linter Rules Reference | Rule | Severity | Description | | ------------------------ | -------- | ---------------------------------- | | `missing-required-field` | warning | Missing name, kind, or type fields | | `duplicate-step-name` | warning | Multiple steps with same name | | `empty-step` | warning | Steps with no executable content | | `unused-variable` | info | Exports never referenced | | `invalid-goto` | warning | Decision goto to non-existent step | | `invalid-depends-on` | warning | Dependency on non-existent step | | `circular-dependency` | warning | Circular step dependencies | The `undefined-variable` rule exists but is not enabled by default due to the large number of built-in variables. Workflows can execute with linter warnings - the linter is designed to help identify potential issues, not block execution. ### Severity Levels * **info** - Best practice suggestions (e.g., unused exports) * **warning** - Potential issues that may cause problems (e.g., duplicate names) * **error** - Critical issues that will likely cause failures ## Workflow Inheritance Workflows can extend parent workflows using the `extends` field. This enables: * Reusing common configurations across workflows * Creating specialized variants (e.g., fast, aggressive, stealth) * Maintaining consistent base workflows with targeted overrides ### Basic Inheritance ```yaml theme={null} kind: module name: subdomain-enum-fast extends: subdomain-enum-base # Override description description: "Fast subdomain enumeration (reduced timeout)" # Override section specifies what to change override: params: threads: default: 50 timeout: default: "10m" ``` ### Extends Field The `extends` field specifies the parent workflow to inherit from: ```yaml theme={null} extends: parent-workflow-name # By name (same directory) extends: ./parent.yaml # By relative path extends: modules/base-module.yaml # By path from workflow directory ``` The child workflow inherits all fields from the parent, with the child's fields taking precedence. ### Override Modes When overriding steps (modules) or modules (flows), you can specify a merge strategy: | Mode | Description | | --------- | ------------------------------------------------------------- | | `replace` | Completely replace parent items with child items | | `prepend` | Add child items before parent items | | `append` | Add child items after parent items (default) | | `merge` | Match by name: replace matching, append new, remove specified | #### Append Mode (Default) ```yaml theme={null} kind: module name: extended-enum extends: base-enum override: steps: mode: append steps: - name: additional-step type: bash command: "extra-tool -t {{Target}}" ``` #### Prepend Mode ```yaml theme={null} override: steps: mode: prepend steps: - name: setup-step type: bash command: "setup-tool" ``` #### Replace Mode ```yaml theme={null} override: steps: mode: replace steps: - name: only-step type: bash command: "completely-different-tool" ``` #### Merge Mode ```yaml theme={null} override: steps: mode: merge # Replace existing step by name replace: - name: subfinder type: bash command: "subfinder -d {{Target}} --all -o {{Output}}/subfinder.txt" # Remove steps by name remove: - amass-passive # Append new steps steps: - name: new-tool type: bash command: "new-tool -t {{Target}}" ``` ### Override Sections The `override` block supports these sections: | Section | Description | | --------------- | ------------------------------------------------------ | | `params` | Override parameter defaults, types, or required status | | `steps` | Override steps (module workflows only) | | `modules` | Override modules (flow workflows only) | | `triggers` | Replace parent triggers entirely | | `dependencies` | Merge with parent dependencies | | `preferences` | Override execution preferences | | `runner_config` | Override runner configuration | | `runner` | Override runner type | ### Parameter Overrides Override specific parameter properties: ```yaml theme={null} override: params: threads: default: 50 # Override default value wordlist: default: "{{Data}}/fast-wordlist.txt" new_param: # Add new parameter default: "value" required: false ``` ### Multi-Level Inheritance Workflows can form inheritance chains: ```yaml theme={null} # base.yaml kind: module name: scan-base steps: - name: common-step type: bash command: "common-tool" # fast.yaml kind: module name: scan-fast extends: scan-base override: params: threads: default: 100 # aggressive-fast.yaml kind: module name: scan-aggressive-fast extends: scan-fast override: params: rate_limit: default: 1000 ``` ### Inheritance Rules 1. **Kind must match** - Child and parent must have the same `kind` (module/flow) 2. **Circular detection** - Circular inheritance chains are detected and rejected 3. **Name uniqueness** - Child's `name` overrides parent's name 4. **File path** - Child's `FilePath` is preserved (for error reporting) ## Best Practices 1. **One task per module** - Keep modules focused 2. **Use flows for pipelines** - Orchestrate with dependencies 3. **Descriptive names** - `subdomain-enum` not `step1` 4. **Document parameters** - Add descriptions 5. **Use tags** - Enable filtering 6. **Use inheritance** - Create base workflows and specialized variants 7. **Prefer merge mode** - For fine-grained step control in child workflows ## Next Steps * [Step Types](step-types) - All step types * [Flows](flows) - Module orchestration * [Variables](variables) - Parameters and exports * [Control Flow](control-flow) - Conditions and routing # Step Types Source: https://docs.osmedeus.org/workflows/step-types Available step types for workflow execution Osmedeus supports 8 step types for different execution needs. ## Overview | Type | Description | Primary Use | | ---------------- | ----------------------- | -------------------------------- | | `bash` | Execute shell commands | Run tools, file operations | | `function` | Run utility functions | Conditions, logging, file checks | | `foreach` | Iterate over file lines | Process lists | | `parallel-steps` | Run steps concurrently | Parallel tool execution | | `remote-bash` | Per-step Docker/SSH | Mixed environments | | `http` | Make HTTP requests | API calls, webhooks | | `llm` | AI-powered processing | Analysis, summarization | | `agent` | Agentic LLM execution | Autonomous tool-calling agents | ## bash Execute shell commands. ### Basic Command ```yaml theme={null} - name: run-subfinder type: bash command: subfinder -d {{target}} -o {{Output}}/subs.txt ``` ### Multiple Commands (Sequential) ```yaml theme={null} - name: setup type: bash commands: - mkdir -p {{Output}}/scans - echo "Starting scan for {{target}}" - date > {{Output}}/start-time.txt ``` ### Parallel Commands ```yaml theme={null} - name: run-tools type: bash parallel_commands: - subfinder -d {{target}} -o {{Output}}/subfinder.txt - amass enum -passive -d {{target}} -o {{Output}}/amass.txt - assetfinder {{target}} > {{Output}}/assetfinder.txt ``` ### Structured Arguments ```yaml theme={null} - name: nuclei-scan type: bash command: nuclei input_args: - name: target-list flag: -l value: "{{Output}}/live.txt" output_args: - name: output flag: -o value: "{{Output}}/nuclei.txt" config_args: - name: templates flag: -t value: "{{Data}}/templates/cves" speed_args: - name: rate-limit flag: -rl value: "150" ``` ### Save Output to File ```yaml theme={null} - name: scan type: bash command: nmap -sV {{target}} std_file: "{{Output}}/nmap-output.txt" ``` ## function Execute utility functions via Goja JavaScript VM. ### Single Function ```yaml theme={null} - name: log-start type: function function: log_info("Starting scan for {{target}}") ``` ### Multiple Functions ```yaml theme={null} - name: check-files type: function functions: - log_info("Checking prerequisites") - fileExists("{{Output}}/targets.txt") - log_info("Ready to proceed") ``` ### Parallel Functions ```yaml theme={null} - name: parallel-checks type: function parallel_functions: - fileLength("{{Output}}/subs.txt") - fileLength("{{Output}}/urls.txt") - fileLength("{{Output}}/live.txt") ``` ### Use in Conditions ```yaml theme={null} - name: run-if-exists type: bash pre_condition: 'fileExists("{{Output}}/targets.txt")' command: nuclei -l {{Output}}/targets.txt ``` ## foreach Iterate over lines in a file with parallel execution using a worker pool. ### Basic Loop ```yaml theme={null} - name: probe-subdomains type: foreach input: "{{Output}}/subdomains.txt" variable: subdomain threads: 10 step: name: httpx-probe type: bash command: echo [[subdomain]] | httpx -silent >> {{Output}}/live.txt ``` ### With Nested Variables ```yaml theme={null} - name: scan-hosts type: foreach input: "{{Output}}/hosts.txt" variable: host threads: 5 step: name: nuclei-scan type: bash command: nuclei -u [[host]] -t {{templates}} -o {{Output}}/nuclei-[[host]].txt ``` ### Bounded Concurrency The foreach executor uses a worker pool pattern: ``` ┌───────────────────────────────────────────────────┐ │ Foreach Executor │ │ │ │ Input File: subdomains.txt (1000 lines) │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────────┐ │ │ │ Worker Pool (threads: 10) │ │ │ │ ┌────┐ ┌────┐ ┌────┐ ... ┌────┐ │ │ │ │ │ W1 │ │ W2 │ │ W3 │ │ W10│ │ │ │ │ └────┘ └────┘ └────┘ └────┘ │ │ │ └────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Results collected after all items processed │ │ │ └───────────────────────────────────────────────────┘ ``` * Workers pull items from a shared queue * Maximum `threads` items processed concurrently * Memory-efficient: doesn't spawn all goroutines upfront * Graceful cancellation on context timeout ### Fields | Field | Required | Description | | ---------------------- | -------- | ----------------------------------------------------------------- | | `input` | Yes | Path to file with items (one per line) | | `variable` | Yes | Variable name for current item | | `threads` | No | Maximum concurrent iterations (default: 1) | | `step` | Yes | Step to execute for each item | | `variable_pre_process` | No | Transform each input line before storing (e.g., `trim([[line]])`) | ### Pre-Processing Input Transform each input line before it is stored in the loop variable: ```yaml theme={null} - name: scan-cleaned-hosts type: foreach input: "{{Output}}/hosts.txt" variable: host variable_pre_process: "trim([[line]])" threads: 10 step: name: probe type: bash command: httpx -u [[host]] -silent ``` ### Variable Syntax Use `[[variable]]` (double brackets) for loop variables to avoid conflicts with `{{templates}}`: ```yaml theme={null} step: name: scan type: bash # [[subdomain]] - replaced per iteration # {{Output}} - resolved once before loop command: nuclei -u [[subdomain]] -o {{Output}}/result-[[subdomain]].txt ``` ### Nested Foreach Foreach steps can contain other foreach steps: ```yaml theme={null} - name: scan-ports-per-host type: foreach input: "{{Output}}/hosts.txt" variable: host threads: 5 step: name: scan-ports type: foreach input: "{{Output}}/ports.txt" variable: port threads: 10 step: name: probe type: bash command: nc -zv [[host]] [[port]] ``` ## parallel-steps Run multiple steps concurrently. ```yaml theme={null} - name: parallel-recon type: parallel-steps parallel_steps: - name: subfinder type: bash command: subfinder -d {{target}} -o {{Output}}/subfinder.txt - name: amass type: bash command: amass enum -passive -d {{target}} -o {{Output}}/amass.txt - name: findomain type: bash command: findomain -t {{target}} -o {{Output}}/findomain.txt ``` Nested steps can be any type: ```yaml theme={null} - name: parallel-checks type: parallel-steps parallel_steps: - name: check-dns type: bash command: dig {{target}} - name: log-check type: function function: log_info("Parallel check running") - name: probe-hosts type: foreach input: "{{Output}}/subs.txt" variable: sub threads: 5 step: type: bash command: echo [[sub]] | httpx ``` ## remote-bash Execute commands in Docker or SSH without module-level runner. ### Docker Execution ```yaml theme={null} - name: docker-nuclei type: remote-bash step_runner: docker step_runner_config: image: projectdiscovery/nuclei:latest volumes: - "{{Output}}:/output" environment: - "API_KEY={{api_key}}" command: nuclei -u {{target}} -o /output/nuclei.txt ``` ### SSH Execution ```yaml theme={null} - name: ssh-nmap type: remote-bash step_runner: ssh step_runner_config: host: "{{ssh_host}}" port: 22 user: "{{ssh_user}}" key_file: ~/.ssh/scanner_key command: nmap -sV {{target}} -oN /tmp/nmap.txt step_remote_file: /tmp/nmap.txt host_output_file: "{{Output}}/nmap-result.txt" ``` ### Fields | Field | Required | Description | | -------------------- | -------- | --------------------------------- | | `step_runner` | Yes | `docker` or `ssh` | | `step_runner_config` | Yes | Runner configuration | | `command` | Yes | Command to execute | | `step_remote_file` | No | Remote file to copy back | | `host_output_file` | No | Local destination for remote file | ## http Make HTTP requests with automatic retries and connection pooling. ### Supported Methods | Method | Description | | -------- | ----------------------- | | `GET` | Retrieve data (default) | | `POST` | Create/submit data | | `PUT` | Update/replace resource | | `PATCH` | Partial update | | `DELETE` | Remove resource | ### GET Request ```yaml theme={null} - name: fetch-api type: http url: "https://api.example.com/data/{{target}}" method: GET headers: Authorization: "Bearer {{api_token}}" exports: api_response: "{{fetch_api_http_resp.response_body}}" status: "{{fetch_api_http_resp.status_code}}" ``` ### POST Request ```yaml theme={null} - name: submit-scan type: http url: "https://scanner.example.com/api/scan" method: POST headers: Content-Type: application/json request_body: | { "target": "{{target}}", "scan_type": "full" } ``` ### PUT Request ```yaml theme={null} - name: update-config type: http url: "https://api.example.com/config/{{target}}" method: PUT headers: Content-Type: application/json Authorization: "Bearer {{api_token}}" request_body: '{"enabled": true}' ``` ### PATCH Request ```yaml theme={null} - name: patch-status type: http url: "https://api.example.com/scan/{{scan_id}}" method: PATCH headers: Content-Type: application/json request_body: '{"status": "completed"}' ``` ### DELETE Request ```yaml theme={null} - name: remove-entry type: http url: "https://api.example.com/entries/{{entry_id}}" method: DELETE headers: Authorization: "Bearer {{api_token}}" ``` ### Auto-Exported Variables After HTTP step execution, variables are exported with the pattern `_http_resp`: ```yaml theme={null} # Access as: {{step_name_http_resp.field}} # Fields available: # status_code - HTTP status code (int) # response_body - Response body (string) # response_headers - Response headers (map) # content_length - Response size in bytes (int) # response_time_ms - Request duration in ms (int) # error - Error message if failed (string or null) # message - Status message (string) ``` ### HTTP Features * **Connection Pooling**: Reuses connections for efficiency * **Automatic Retries**: Retries on network errors and 5xx responses (up to 3 attempts) * **Timeout**: Configurable via step `timeout` field (default: 30s) * **Template Support**: Headers and request body support `{{variable}}` interpolation ## llm AI-powered processing using LLM APIs (OpenAI-compatible). ### Chat Completion ```yaml theme={null} - name: analyze-findings type: llm messages: - role: system content: You are a security analyst. Analyze the findings and provide a summary. - role: user content: | Analyze these vulnerability findings: {{readFile("{{Output}}/vulnerabilities.txt")}} exports: analysis: "{{analyze_findings_content}}" ``` ### Message Roles | Role | Description | | ----------- | ------------------------------- | | `system` | System prompt defining behavior | | `user` | User input/question | | `assistant` | Model's previous response | | `tool` | Tool/function call result | ### With Tool Calling Define tools the LLM can invoke (OpenAI-compatible function calling): ```yaml theme={null} - name: intelligent-scan type: llm messages: - role: system content: You are a security scanner assistant. - role: user content: Analyze {{target}} and suggest next steps. tools: - type: function function: name: run_scan description: Execute a security scan parameters: type: object properties: scan_type: type: string enum: [port, vuln, web] target: type: string description: Target to scan required: - scan_type - target tool_choice: auto # auto, none, or {"type": "function", "function": {"name": "run_scan"}} ``` Tool calls are available in exports as `{{step_name_llm_resp.tool_calls}}`. ### Embeddings Generate vector embeddings for text: ```yaml theme={null} - name: generate-embeddings type: llm is_embedding: true embedding_input: - "{{readFile('{{Output}}/finding1.txt')}}" - "{{readFile('{{Output}}/finding2.txt')}}" exports: embeddings: "{{generate_embeddings_llm_resp.embeddings}}" ``` ### Multimodal Content (Vision) Include images in messages: ```yaml theme={null} - name: analyze-screenshot type: llm messages: - role: user content: - type: text text: "Analyze this application screenshot for security issues" - type: image_url image_url: url: "{{Output}}/screenshot.png" ``` ### Structured Output (JSON Schema) Force structured JSON responses: ```yaml theme={null} - name: structured-analysis type: llm messages: - role: system content: You are a security analyst. - role: user content: Analyze {{target}} and return structured findings. llm_config: response_format: type: json_schema json_schema: name: security_findings schema: type: object properties: severity: type: string enum: [critical, high, medium, low, info] findings: type: array items: type: object properties: title: type: string description: type: string required: - severity - findings ``` ### Configuration Override Override global LLM settings per step: ```yaml theme={null} - name: custom-llm type: llm llm_config: model: gpt-4-turbo max_tokens: 4000 temperature: 0.3 top_p: 0.95 timeout: 5m max_retries: 5 custom_headers: X-Custom-Header: value messages: - role: user content: Analyze {{target}} ``` ### Auto-Exported Variables After LLM step execution: ```yaml theme={null} # Access as: {{step_name_llm_resp.field}} or {{step_name_content}} # Fields available in step_name_llm_resp: # id - Response ID # model - Model used # content - Response text (first choice) # finish_reason - Why generation stopped # role - Message role # tool_calls - Tool calls if any (array) # all_contents - All choices if n > 1 # usage - Token usage (prompt_tokens, completion_tokens, total_tokens) # embeddings - Embedding vectors (for embedding mode) # # Shorthand: {{step_name_content}} directly contains the response text ``` ### Provider Rotation If multiple LLM providers are configured, the executor automatically: * Rotates to next provider on rate limits or errors * Retries up to `max_retries * provider_count` times * Records rate limit metrics for monitoring ## agent Agentic LLM execution with an autonomous tool-calling loop. The agent receives a task, plans its approach, calls tools iteratively, and produces a final answer. ### Basic Usage ```yaml theme={null} - name: analyze-target type: agent query: "Enumerate subdomains of {{Target}} and summarize findings." system_prompt: "You are a security reconnaissance agent." max_iterations: 10 agent_tools: - preset: bash - preset: read_file - preset: save_content exports: findings: "{{agent_content}}" ``` ### Preset Tools The following preset tools are available via the `preset` field: | Preset | Description | | ------------------ | ------------------------------------ | | `bash` | Execute shell commands | | `read_file` | Read entire file contents | | `read_lines` | Read specific line range from a file | | `file_exists` | Check if a file exists | | `file_length` | Count non-empty lines in a file | | `append_file` | Append content to a file | | `save_content` | Write content to a file | | `glob` | Find files matching a glob pattern | | `grep_string` | Search for literal string in files | | `grep_regex` | Search for regex pattern in files | | `http_get` | Make HTTP GET request | | `http_request` | Make HTTP request (any method) | | `jq` | Query JSON with jq expressions | | `exec_python` | Execute inline Python code | | `exec_python_file` | Execute a Python file | | `exec_ts` | Execute inline TypeScript code | | `exec_ts_file` | Execute a TypeScript file | | `run_module` | Run an Osmedeus module | | `run_flow` | Run an Osmedeus flow | ```yaml theme={null} agent_tools: - preset: bash - preset: read_file - preset: grep_regex - preset: save_content - preset: http_get ``` ### Custom Tool Handlers Define custom tools with a handler expression: ```yaml theme={null} agent_tools: - preset: bash - name: lookup_whois description: "Look up WHOIS information for a domain" parameters: type: object properties: domain: type: string description: "Domain to query" required: [domain] handler: "exec('whois ' + args.domain)" ``` ### Multi-Goal Queries Use `queries` for multiple goals evaluated in sequence: ```yaml theme={null} - name: multi-task type: agent queries: - "Enumerate subdomains of {{Target}}" - "Probe discovered hosts for HTTP services" - "Summarize all findings" max_iterations: 20 agent_tools: - preset: bash - preset: save_content ``` ### Sub-Agents Spawn specialized sub-agents from the main agent: ```yaml theme={null} - name: coordinator type: agent query: "Perform full reconnaissance of {{Target}}" max_iterations: 15 agent_tools: - preset: bash - preset: read_file sub_agents: - name: dns-expert description: "DNS enumeration specialist" system_prompt: "You are a DNS enumeration expert." agent_tools: - preset: bash - preset: save_content max_iterations: 5 - name: web-scanner description: "Web application scanner" system_prompt: "You are a web security scanner." agent_tools: - preset: bash - preset: http_get max_iterations: 5 ``` The main agent can invoke sub-agents via the auto-generated `spawn_agent` tool. ### Memory Configuration Control conversation memory for long-running agents: ```yaml theme={null} - name: long-task type: agent query: "Perform deep analysis of {{Target}}" max_iterations: 50 memory: max_messages: 30 summarize_on_truncate: true persist_path: "{{Output}}/agent/conversation.json" resume_path: "{{Output}}/agent/conversation.json" agent_tools: - preset: bash - preset: save_content ``` | Field | Description | | ----------------------- | -------------------------------------------- | | `max_messages` | Sliding window size for conversation history | | `summarize_on_truncate` | Summarize old messages before removing them | | `persist_path` | Save conversation to file after completion | | `resume_path` | Resume from a previous conversation file | ### Planning Stage Run a planning prompt before the main execution loop: ```yaml theme={null} - name: planned-recon type: agent query: "Execute the reconnaissance plan for {{Target}}" plan_prompt: "Create a step-by-step reconnaissance plan for {{Target}}. Consider subdomain enumeration, port scanning, and service fingerprinting." max_iterations: 15 agent_tools: - preset: bash - preset: save_content exports: plan: "{{agent_plan}}" results: "{{agent_content}}" ``` ### Structured Output Enforce a JSON schema on the agent's final output: ```yaml theme={null} - name: structured-recon type: agent query: "Analyze {{Target}} and return structured findings." max_iterations: 10 output_schema: type: object properties: target: type: string subdomains: type: array items: type: string open_ports: type: array items: type: integer summary: type: string required: [target, subdomains, summary] agent_tools: - preset: bash - preset: read_file ``` ### Model Selection Specify preferred models (tried in order before falling back to default): ```yaml theme={null} - name: smart-agent type: agent query: "Analyze {{Target}}" models: - claude-sonnet-4-20250514 - gpt-4-turbo max_iterations: 10 agent_tools: - preset: bash ``` ### Tool Tracing Hooks Add JavaScript hooks for tool call monitoring: ```yaml theme={null} - name: traced-agent type: agent query: "Scan {{Target}}" max_iterations: 10 on_tool_start: "log_info('Calling tool: ' + tool_name)" on_tool_end: "log_info('Tool ' + tool_name + ' returned: ' + tool_result.substring(0, 100))" agent_tools: - preset: bash - preset: save_content ``` ### Stop Condition Evaluate a JS expression after each iteration to stop early: ```yaml theme={null} - name: conditional-agent type: agent query: "Find vulnerabilities in {{Target}}" max_iterations: 20 stop_condition: "file_exists('{{Output}}/critical-finding.txt')" agent_tools: - preset: bash - preset: save_content ``` ### Parallel Tool Calls Control whether the agent can execute multiple tool calls in parallel (enabled by default): ```yaml theme={null} - name: sequential-agent type: agent query: "Carefully analyze {{Target}}" max_iterations: 10 parallel_tool_calls: false # Force sequential tool execution agent_tools: - preset: bash ``` ### Auto-Exported Variables After agent step execution, these variables are automatically available: | Variable | Description | | ----------------------------- | ------------------------------------------------ | | `{{agent_content}}` | Final agent response text | | `{{agent_history}}` | Full conversation history (JSON) | | `{{agent_iterations}}` | Number of iterations used | | `{{agent_total_tokens}}` | Total tokens consumed | | `{{agent_prompt_tokens}}` | Prompt tokens consumed | | `{{agent_completion_tokens}}` | Completion tokens consumed | | `{{agent_tool_results}}` | All tool call results (JSON) | | `{{agent_plan}}` | Planning stage output (if `plan_prompt` was set) | | `{{agent_goal_results}}` | Per-goal results (for multi-goal `queries`) | ### Fields Reference | Field | Required | Description | | --------------------- | -------- | ---------------------------------------------------------------------------------------------- | | `query` | Yes\* | Task prompt for the agent | | `queries` | Yes\* | Multiple task prompts (alternative to `query`) | | `max_iterations` | Yes | Maximum tool-calling loop iterations (must be > 0) | | `agent_tools` | No | List of preset or custom tools | | `system_prompt` | No | System prompt for the agent | | `sub_agents` | No | Inline sub-agents spawnable via `spawn_agent` tool | | `memory` | No | Sliding window config (`max_messages`, `summarize_on_truncate`, `persist_path`, `resume_path`) | | `models` | No | Preferred models tried in order before falling back to default | | `output_schema` | No | JSON schema enforced on final output | | `plan_prompt` | No | Planning stage prompt run before the main loop | | `stop_condition` | No | JS expression evaluated after each iteration | | `on_tool_start` | No | JS hook expression run before each tool call | | `on_tool_end` | No | JS hook expression run after each tool call | | `parallel_tool_calls` | No | Enable/disable parallel tool execution (default: true) | \* Either `query` or `queries` is required. ## Common Step Fields All steps support these fields: ```yaml theme={null} - name: step-name # Required: unique name type: bash # Required: step type depends_on: # Step dependencies (DAG execution) - previous-step-1 - previous-step-2 pre_condition: 'expr' # Skip if false timeout: 30m # Step timeout (e.g., 30s, 30m, 1h, 1d) log: "{{Output}}/step.log" # Log file path exports: # Export values var_name: "{{value}}" on_success: # Success handlers - action: log message: "Done" on_error: # Error handlers - action: continue decision: # Conditional routing switch: "{{value}}" cases: "match": { goto: other-step } default: { goto: fallback } ``` ### Field Reference | Field | Description | | --------------- | -------------------------------------------------------------------------------------------- | | `name` | Required. Unique step identifier | | `type` | Required. Step type (bash, function, foreach, parallel-steps, remote-bash, http, llm, agent) | | `depends_on` | Array of step names this step depends on (enables parallel execution) | | `pre_condition` | Expression to evaluate; step skipped if false | | `timeout` | Maximum execution time (default varies by step type) | | `log` | Path to write step output log | | `exports` | Map of variable names to values to export | | `on_success` | Actions to execute on successful completion | | `on_error` | Actions to execute on failure | | `decision` | Conditional routing based on variable values | ## Next Steps * [Variables](variables) - Exports and propagation * [Control Flow](control-flow) - Conditions and routing * [Functions Reference](../functions/reference) - Available functions # Variables and Exports Source: https://docs.osmedeus.org/workflows/variables Data flow and parameter management Manage data flow between steps using parameters and exports. ## Parameters ### Defining Parameters ```yaml theme={null} params: - name: target required: true description: Target domain - name: threads default: "10" description: Thread count - name: wordlist default: "{{Data}}/wordlists/common.txt" ``` ### Using Parameters ```yaml theme={null} steps: - name: scan type: bash command: subfinder -d {{target}} -t {{threads}} -w {{wordlist}} ``` ### Passing Parameters ```bash theme={null} # Single parameter osmedeus run -m scan -t example.com -p 'threads=20' # Multiple parameters osmedeus run -m scan -t example.com -p 'threads=20' -p 'wordlist=/path/list.txt' # From file osmedeus run -m scan -t example.com -P params.yaml ``` ## Exports Exports pass values from one step to the next. ### Basic Export ```yaml theme={null} steps: - name: count-lines type: bash command: wc -l {{Output}}/hosts.txt | cut -d' ' -f1 exports: host_count: "{{stdout}}" - name: log-count type: function function: log_info("Found {{host_count}} hosts") ``` ### Export Sources | Source | Description | | ------------------------ | ----------------------- | | `{{stdout}}` | Command standard output | | `{{stderr}}` | Command standard error | | `{{exit_code}}` | Command exit code | | `{{http_status_code}}` | HTTP response status | | `{{http_response_body}}` | HTTP response body | | `{{llm_response}}` | LLM chat response | ### HTTP Exports ```yaml theme={null} - name: api-call type: http url: "https://api.example.com/data" method: GET exports: response_data: "{{http_response_body}}" status: "{{http_status_code}}" - name: process type: function function: log_info("Status: {{status}}, Data: {{response_data}}") ``` ### Agent Exports ```yaml theme={null} - name: analyze type: agent query: "Analyze {{Target}} for vulnerabilities" max_iterations: 10 agent_tools: - preset: bash - preset: read_file exports: findings: "{{agent_content}}" token_usage: "{{agent_total_tokens}}" ``` | Source | Description | | ----------------------------- | ------------------------------------------------ | | `{{agent_content}}` | Final agent response text | | `{{agent_history}}` | Full conversation history (JSON) | | `{{agent_iterations}}` | Number of iterations used | | `{{agent_total_tokens}}` | Total tokens consumed | | `{{agent_prompt_tokens}}` | Prompt tokens consumed | | `{{agent_completion_tokens}}` | Completion tokens consumed | | `{{agent_tool_results}}` | All tool call results (JSON) | | `{{agent_plan}}` | Planning stage output (if `plan_prompt` was set) | | `{{agent_goal_results}}` | Per-goal results (for multi-goal `queries`) | ### Function Exports ```yaml theme={null} - name: read-file type: function function: readFile("{{Output}}/data.txt") exports: file_content: "{{result}}" - name: use-content type: bash command: echo "Content: {{file_content}}" ``` ## Variable Scope ### Step-Level Scope Exports are available to all subsequent steps: ```yaml theme={null} steps: - name: step1 type: bash command: echo "value1" exports: var1: "{{stdout}}" - name: step2 type: bash command: echo "value2" exports: var2: "{{stdout}}" - name: step3 type: bash command: echo "{{var1}} and {{var2}}" # Both available ``` ### Foreach Variable Scope Loop variables use `[[]]` syntax and are only available inside the loop: ```yaml theme={null} - name: process-hosts type: foreach input: "{{Output}}/hosts.txt" variable: host step: name: scan type: bash command: nmap [[host]] -o {{Output}}/nmap-[[host]].txt # [[host]] = current iteration value # {{Output}} = regular template variable ``` ## Resolution Order Variables are resolved in this order: 1. **Exports** from previous steps 2. **Parameters** from user input 3. **Built-in variables** (Target, Output, etc.) 4. **Environment variables** ## Built-in Variables Osmedeus provides a comprehensive set of built-in variables that are automatically available in all workflows. These variables are recognized by the linter and do not need to be defined. ### Path Variables | Variable | Description | | -------------------------- | ------------------------------------ | | `{{BaseFolder}}` | Osmedeus installation directory | | `{{Binaries}}` | Path to tool binaries | | `{{Data}}` | Path to data files | | `{{ExternalData}}` | Path to external data files | | `{{ExternalConfigs}}` | Path to external configuration files | | `{{ExternalAgentConfigs}}` | Path to agent configuration files | | `{{ExternalAgents}}` | Path to agent scripts | | `{{ExternalScripts}}` | Path to external scripts | | `{{Workflows}}` | Path to workflows directory | | `{{MarkdownTemplates}}` | Path to markdown templates | | `{{ExternalMarkdowns}}` | Path to external markdown files | | `{{SnapshotsFolder}}` | Path to snapshots storage | | `{{Workspaces}}` | Path to workspaces directory | ### Target Variables | Variable | Description | | ----------------- | ------------------------------------------- | | `{{Target}}` | Current scan target | | `{{target}}` | Current scan target (lowercase alias) | | `{{TargetFile}}` | Path to target file (for multi-target runs) | | `{{TargetSpace}}` | Sanitized target (filesystem safe) | ### Output Variables | Variable | Description | | --------------- | -------------------------------------------- | | `{{Output}}` | Workspace output directory | | `{{output}}` | Workspace output directory (lowercase alias) | | `{{Workspace}}` | Workspace directory | | `{{workspace}}` | Workspace directory (lowercase alias) | ### Thread Variables | Variable | Description | | ----------------- | ------------------------------ | | `{{threads}}` | Thread count (based on tactic) | | `{{Threads}}` | Thread count (uppercase alias) | | `{{baseThreads}}` | Base thread count | ### Metadata Variables | Variable | Description | | ------------------ | --------------------------------------------------- | | `{{Version}}` | Osmedeus version | | `{{TaskDate}}` | Task date | | `{{TaskID}}` | Unique task identifier | | `{{TimeStamp}}` | Unix timestamp | | `{{CurrentTime}}` | Current time | | `{{Today}}` | Current date (YYYY-MM-DD) | | `{{RandomString}}` | Random 6-character string | | `{{ModuleName}}` | Current module name | | `{{FlowName}}` | Parent flow name (empty if running module directly) | | `{{RunUUID}}` | Unique run identifier (UUID) | | `{{DBRunID}}` | Database run ID | ### State File Variables | Variable | Description | | ------------------------- | ------------------------------ | | `{{StateExecutionLog}}` | Path to execution log | | `{{StateConsoleLog}}` | Path to console log | | `{{StateCompletedFile}}` | Path to completion marker file | | `{{StateFile}}` | Path to state file | | `{{StateWorkflowFile}}` | Path to workflow state file | | `{{StateWorkflowFolder}}` | Path to workflow state folder | ### Heuristic Variables These variables are populated by automatic target analysis: | Variable | Description | | ------------------------- | ----------------------------------- | | `{{TargetType}}` | Detected target type | | `{{TargetRootDomain}}` | Root domain extracted from target | | `{{TargetTLD}}` | Top-level domain | | `{{TargetSLD}}` | Second-level domain | | `{{Org}}` | Organization (if detected) | | `{{TargetBaseURL}}` | Base URL of target | | `{{TargetRootURL}}` | Root URL of target | | `{{TargetHostname}}` | Hostname from target URL | | `{{TargetHost}}` | Host from target | | `{{TargetPort}}` | Port from target URL | | `{{TargetPath}}` | Path from target URL | | `{{TargetFileExt}}` | File extension from target URL | | `{{TargetScheme}}` | URL scheme (http/https) | | `{{TargetIsWildcard}}` | Whether target is a wildcard | | `{{TargetResolvedIP}}` | Resolved IP address | | `{{TargetStatusCode}}` | HTTP status code from target | | `{{TargetContentLength}}` | Content length from target response | | `{{HeuristicsCheck}}` | Result of heuristics analysis | ### Platform Variables Automatically detected environment information: | Variable | Description | | --------------------------- | ---------------------------------------------------- | | `{{PlatformOS}}` | Operating system (`linux`, `darwin`, `windows`) | | `{{PlatformArch}}` | CPU architecture (`amd64`, `arm64`) | | `{{PlatformInDocker}}` | `"true"` if running inside a Docker container | | `{{PlatformInKubernetes}}` | `"true"` if running inside a Kubernetes pod | | `{{PlatformCloudProvider}}` | Cloud provider name (`aws`, `gcp`, `azure`, `local`) | ```yaml theme={null} steps: - name: platform-check type: bash pre_condition: '{{PlatformOS}} == "linux"' command: linux-specific-tool {{Target}} ``` ### Event Variables Available when a workflow is triggered by an event: | Variable | Description | | -------------------- | -------------------------------- | | `{{EventEnvelope}}` | Full event envelope (JSON) | | `{{EventTopic}}` | Event topic (e.g., `assets.new`) | | `{{EventSource}}` | Source of the event | | `{{EventDataType}}` | Type of event data | | `{{EventTimestamp}}` | Event timestamp | ### Chunk Variables Used for parallel processing of large inputs: | Variable | Description | | ----------------- | ----------------------------- | | `{{ChunkIndex}}` | Current chunk index | | `{{ChunkSize}}` | Size of each chunk | | `{{TotalChunks}}` | Total number of chunks | | `{{ChunkStart}}` | Start offset of current chunk | | `{{ChunkEnd}}` | End offset of current chunk | ```yaml theme={null} params: - name: threads default: "10" steps: - name: first type: bash command: echo "20" exports: threads: "{{stdout}}" # Export named 'threads' - name: second type: bash command: run -t {{threads}} # Uses export (20), not param (10) ``` ## Nested Variables Variables can contain other variables: ```yaml theme={null} params: - name: scan_type default: "basic" - name: output_path default: "{{Output}}/{{scan_type}}" steps: - name: scan type: bash command: scan -o {{output_path}}/results.txt # Resolves to: {{Output}}/basic/results.txt ``` ## Generator Functions Generate dynamic values for parameters using the `generator` field: ```yaml theme={null} params: - name: scan_id generator: "uuid()" - name: timestamp generator: "currentTimestamp()" - name: user generator: "getEnvVar('USER', 'unknown')" - name: run_date generator: "currentDate()" ``` The `generator` field is evaluated at workflow load time to produce the parameter value. ### Available Generators | Generator | Description | Example | | ------------------------------ | ---------------------------------------- | ------------------------------ | | `uuid()` | UUID v4 | `a1b2c3d4-...` | | `currentDate(format?)` | Current date (default: YYYY-MM-DD) | `2026-02-17` | | `currentTimestamp()` | Unix timestamp | `1739808000` | | `getEnvVar(key, default?)` | Environment variable | `getEnvVar('USER', 'unknown')` | | `concat(str1, str2, ...)` | Concatenate strings | `concat('scan-', 'target')` | | `randomInt(min?, max?)` | Random integer (default: 0-100) | `randomInt(1, 1000)` | | `randomString(length?)` | Random alphanumeric string (default: 16) | `randomString(8)` | | `execCmd(command)` | Execute shell command | `execCmd('whoami')` | | `toLower(str)` | Convert to lowercase | `toLower('ABC')` | | `toUpper(str)` | Convert to uppercase | `toUpper('abc')` | | `trim(str)` | Trim whitespace | `trim(' hello ')` | | `replace(str, old, new)` | Replace occurrences | `replace('a-b', '-', '_')` | | `split(str, delim, index?)` | Split and get element | `split('a,b,c', ',', 1)` | | `join(delim, str1, str2, ...)` | Join strings with delimiter | `join('-', 'a', 'b')` | ## Flow Variable Propagation ### Flow to Module ```yaml theme={null} # Flow kind: flow params: - name: target - name: threads default: "50" modules: - name: scan path: modules/scan.yaml params: threads: "{{threads}}" # Pass flow param to module ``` ### Module Exports in Flow Module exports are not automatically available to other modules. Use shared output files: ```yaml theme={null} # Module A writes command: subfinder -d {{target}} -o {{Output}}/subs.txt # Module B reads (depends_on: [A]) command: httpx -l {{Output}}/subs.txt ``` ## Common Patterns ### Chained Processing ```yaml theme={null} steps: - name: enumerate type: bash command: subfinder -d {{target}} -silent exports: raw_subs: "{{stdout}}" - name: filter type: function function: | split("{{raw_subs}}", "\n") .filter(s => s.endsWith(".{{target}}")) .join("\n") exports: filtered_subs: "{{result}}" - name: save type: bash command: echo "{{filtered_subs}}" > {{Output}}/subs.txt ``` ### Conditional on Export ```yaml theme={null} steps: - name: count type: bash command: wc -l < {{Output}}/hosts.txt exports: count: "{{stdout}}" - name: scan-if-hosts type: bash pre_condition: '{{count}} > 0' command: nuclei -l {{Output}}/hosts.txt ``` ### Environment Variables ```yaml theme={null} params: - name: api_key default: "{{getEnvVar('API_KEY', '')}}" steps: - name: api-call type: http url: "https://api.example.com/scan" headers: Authorization: "Bearer {{api_key}}" ``` ## Best Practices 1. **Use descriptive export names** ```yaml theme={null} exports: subdomain_count: "{{stdout}}" # Good x: "{{stdout}}" # Bad ``` 2. **Document parameter meanings** ```yaml theme={null} params: - name: severity default: "high,critical" description: Nuclei severity filter ``` 3. **Provide sensible defaults** ```yaml theme={null} params: - name: threads default: "10" # Works without explicit parameter ``` 4. **Use files for large data** ```yaml theme={null} # Good: write to file command: subfinder -d {{target}} -o {{Output}}/subs.txt # Avoid: large stdout exports exports: all_subs: "{{stdout}}" # Could be huge ``` ## Next Steps * [Control Flow](control-flow) - Using exports in conditions * [Templates](../concepts/templates) - Template syntax * [Functions Reference](../functions/reference) - Available functions