Understand the workflow¶
Workflow is the core of the Osmedeus Engine which represents your methodology as YAML files.
All Workflow files are YAML-based so make sure you follow the YAML syntax. Otherwise, it wouldn't work
- Module contains detail of multiple step.
- Flow contains multiple module and also define order how to run these modules.
- Step is smallest part of the Osmedeus routine.
Example Flow¶
General flow¶
name: general
desc: run normal routine
type: general # this is a folder name that will contains module file
validator: domain # validate the input provide from -t option
routines:
- modules:
- subdomain
- modules:
- probing
- modules:
- ssame
- modules:
- screenshot
- modules: # these modules will be run in parallel
- fingerprint
- spider
- sto
- modules: # these modules will be run in parallel
- archive
- ipspace
- modules:
- vulnscan
- modules:
- vhostscan
- modules:
- portscan
- modules:
- pdirbscan
- modules:
- dirbscan
# push final result again
- modules:
- summary
Flow with custom parameters¶
name: gently-extensive
desc: run extensive routine but with very low threads
type: general
validator: domain
force-params: true
params:
- subfinderThreads: "20"
# probing
- dnsThreads: "300"
- httpThreads: "30"
- massdnsRateBrute: "300"
# screenshot
- screenThreads: "8"
# fingerprint & spider
- ssthreads: "30"
- overviewThreads: '30'
- stoThreads: '30'
- spiderThreads: "5"
# vulnscan
- nucleiThreads: "30"
- jaelesThreads: "20"
# dirbscan
- ffThreads: '20' # threads for single ffuf-scan
- dirbThreads: '2' # how many ffuf run at the same time
# portscan
- ports: "0-65535"
- rateRustScan: "5000"
- nmapThreads: "10"
routines:
- flow: 'extensive'
modules:
- subdomain
- flow: 'extensive'
modules:
- probing
- flow: 'extensive'
modules:
- ssame
- modules:
- screenshot
- modules:
- sto
- fingerprint
- flow: 'extensive'
modules:
- spider
- archive
- modules:
- ipspace
- modules:
- vulnscan
# - modules:
# - vhostscan
- modules:
- portscan
- flow: 'extensive'
modules:
- port-fingerprint
- pvulnscan
- flow: 'extensive'
modules:
- dirbscan
# push final result again
- modules:
- summary
Example Modules¶
subdomain module¶
name: subdomain
desc: Scanning for subdomain
report:
final:
- "{{Output}}/subdomain/final-{{Workspace}}.txt"
- "{{Output}}/subdomain/more-{{Workspace}}.txt"
# {{Output}} == {{Workspaces}} + {{Workspace}} but strip "/" char
pre_run:
- CreateFolder("{{Storages}}/subdomain/{{Workspace}}/")
- CreateFolder("{{Storages}}/summary/{{Workspace}}/")
- CreateFolder("{{Output}}/subdomain/")
params:
- subthreads: "50"
- amassTimeout: "3h"
- amassConfig: "{{Data}}/amass-config/basic-config.yaml"
steps:
- required:
- "{{Binaries}}/amass"
- "{{Binaries}}/subfinder"
- "{{Binaries}}/assetfinder"
- "{{Binaries}}/findomain"
commands: # these two commands will run in parallels
- "timeout -k 1m {{amassTimeout}} {{Binaries}}/amass enum -config {{amassConfig}} -silent -nocolor -d {{Target}} -o {{Output}}/subdomain/{{Workspace}}-raw-amass.txt > /dev/null 2>&1"
- "{{Binaries}}/assetfinder -subs-only {{Target}} > {{Output}}/subdomain/{{Workspace}}-assetfinder.txt"
# these two commands will run in parallels
- commands:
- "{{Binaries}}/findomain -u {{Output}}/subdomain/{{Workspace}}-findomain.txt -t {{Target}} > /dev/null 2>&1"
- "{{Binaries}}/subfinder -d {{Target}} -t {{subthreads}} -o {{Output}}/subdomain/{{Workspace}}-subfinder.txt > /dev/null 2>&1"
# cleaning some result
- scripts:
- Append("{{Output}}/subdomain/sum-{{Workspace}}.txt", "{{Output}}/subdomain/{{Workspace}}-amass.txt")
- Append("{{Output}}/subdomain/sum-{{Workspace}}.txt", "{{Output}}/subdomain/{{Workspace}}-subfinder.txt")
- Append("{{Output}}/subdomain/sum-{{Workspace}}.txt", "{{Output}}/subdomain/{{Workspace}}-assetfinder.txt")
- Append("{{Output}}/subdomain/sum-{{Workspace}}.txt", "{{Output}}/subdomain/{{Workspace}}-findomain.txt")
# remove junk subdomain like sample@subdomain.com and 1-2-3.subdomain.com format
- ExecCmd("cat {{Output}}/subdomain/sum-{{Workspace}}.txt | {{Binaries}}/cleansub -t '{{Target}}' > {{Output}}/subdomain/final-{{Workspace}}.txt")
- scripts:
- SortU("{{Output}}/subdomain/final-{{Workspace}}.txt")
# get more related domains
- required:
- "{{Binaries}}/metabigor"
# assume you run 'osmedeus scan -t example.com' then {{Org}} == 'example'
commands:
- "echo '{{Org}}' | {{Binaries}}/metabigor cert --json -o {{Output}}/subdomain/more-json-{{Workspace}}.txt"
scripts:
- ExecCmd("cat {{Output}}/subdomain/more-json-{{Workspace}}.txt | jq -r '.Domain' | sed 's/\*.//g' | sort -u > {{Output}}/subdomain/more-{{Workspace}}.txt")
post_run:
# delete all files in workspaces folder except a file lists in report section
- Cleaning("{{Output}}/subdomain/")
dirbscan module¶
name: dirbscan
desc: Run Dirbscan
report:
final:
- "{{Output}}/directory/beautify-{{Workspace}}.txt"
- "{{Output}}/directory/beautify-{{Workspace}}.csv"
params:
- httpFile: "{{Output}}/probing/http-{{Workspace}}.txt"
- wordlists: "{{Data}}/wordlists/content/small.txt"
- lines: "20"
- ffThreads: '20' # threads for single site
- dirbThreads: '10'
- dlimit: '50000'
- recursion: '0'
- commitLength: '400'
- chan: '#mics'
- ffTimeout: '2h'
- defaultUA: "User-Agent: Mozilla/5.0 (compatible; Osmedeus/v4; +https://github.com/j3ssie/osmedeus)"
pre_run:
- CreateFolder("{{Output}}/directory")
steps:
# check if the size is too big, We don't want tons of garbage here
- conditions:
- "FileLength('{{httpFile}}') > {{dlimit}}"
scripts:
- ErrPrintf("Filter", "Got input file greater than {{dlimit}} line")
- Exit(1)
- required:
- "{{Binaries}}/ffuf"
- "{{httpFile}}"
source: "{{httpFile}}"
threads: '{{dirbThreads}}'
commands:
- "{{Binaries}}/ffuf -t {{ffThreads}} -H '{{defaultUA}}' -s -timeout 15 -ac -fc '429,403,404' -D -e 'asp,aspx,pl,php,html,htm,jsp,cgi' -of json -o {{Output}}/directory/raw-[[._id_]].json -u '[[.line]]/FUZZ' -w {{wordlists}}:FUZZ >/dev/null 2>&1"
# clean up and generate beautify report
- scripts:
- ExecCmd("awk '{print}' {{Output}}/directory/raw-*.json > {{Output}}/directory/summary-json-{{Workspace}}.txt")
- CleanFFUFJson("{{Output}}/directory/summary-json-{{Workspace}}.txt", "{{Output}}/directory/beautify-{{Workspace}}.csv")
- ExecCmd("cat {{Output}}/directory/beautify-{{Workspace}}.csv | {{Binaries}}/csvtk pretty --no-header-row -I -s ' | ' -W 75 > {{Output}}/directory/beautify-{{Workspace}}.txt")
- TeleMessByFile("#dirb", "{{Output}}/directory/beautify-{{Workspace}}.txt")
- Cat('{{Output}}/directory/beautify-{{Workspace}}.txt')
post_run:
- TotalDirb("{{Output}}/directory/beautify-{{Workspace}}.txt")
Step¶
Here are three different steps for running commands and scripts.
Normal Step¶
steps: # all the steps will run in serial
- commands: # unix command and it will be run in parallel
- "unix command 1 here"
- "unix command 2 here"
scripts:
- ExecCmd("unix command 3 heret") # command 3 and 4 below it will be run in serial
- ExecCmd("unix command 4 here")
- SortU("filename-here.txt")
Step with conditions¶
steps: # all the steps will run in serial
- required: # Check if all the files exist or the step will not run
- filename-1-here.txt
conditions: # Boolean conditions check with built-in scripts
- "FileLength('filename.txt') > 10000"
- Exit(1) # this will exit the module imeediately
commands: # unix command and it will be run in parallel
- "unix command 1 here"
- "unix command 2 here"
scripts:
- ExecCmd("unix command 3 heret") # command 3 and 4 below it will be run in serial
- ExecCmd("unix command 4 here")
- SortU("filename-here.txt")
- TeleMessByFile("#dirb", "beautify-{{Workspace}}.txt")
- required: # Check if all the files exist or the step will not run
- filename-2-here.txt
conditions: # Boolean conditions check with built-in scripts
- "FileLength('filename-2-here.txt') > 10000"
## only run if conditions is false
rcommands: # run in parallel
- "unix command 1 here"
- "unix command 2 here"
rscripts:
- ErrPrintf("Filter", "Got input file greater than 1000 line")
- Exit(1) # this will exit the module imeediately
Step with the loop using source:
tag and [[.line]]
variable.¶
steps:
# variation 2 that will run the step but with input is each like of 'source' section
- source: "{{inputFile}}" # source file to loop through
threads: '{{dirbThreads}}'
commands: # {{Binaries}} is the path to binaries which usually ~/osmedeus/binaries/ but you can use any tool inside your $PATH environment variable
- "{{Binaries}}/ffuf-mod -H 'X-Forwarded-For: 127.0.0.1' -t {{fthreads}} -recursion-depth {{recursion}} -D -e 'asp,aspx,php,html,htm,jsp,cgi' -timeout 15 -get-hash -ac -s -fc '429,404,400' -of json -o {{Output}}/directory/raw-[[._id_]].json -u '[[.line]]/FUZZ' -w {{wordlists}}:FUZZ"
scripts:
- SortU("{{Storages}}/paths/{{Workspace}}/paths-{{Workspace}}.csv")