WhatWeb：下一代网站指纹技术栈识别扫描器，拥有超过1800个扩展插件

发表于 2019年07月04日
安全工具

WhatWeb介绍

WhatWeb可识别Web技术，包括指纹识别、内容管理系统（CMS）、博客平台、统计/分析包、JavaScript库、Web服务器和嵌入式设备。WhatWeb有超过1800个插件，每个插件都能识别不同的东西。WhatWeb还标识版本号，电子邮件地址，帐户ID，Web框架模块，SQL错误等。

WhatWeb支持攻击级别的设置来控制速度和稳定性。默认的攻击级别称为“隐身”，速度最快，只需要一个HTTP请求。适用于扫描公共网站。WhatWeb还开发了更高效的模式用于渗透测试。

WhatWeb截图

WhatWeb识别网站指纹
WhatWeb技术栈识别

WhatWeb特点

超过1800个插件
控制速度/隐身和可靠性之间的权衡
性能调整。控制同时扫描多少个网站。
多种日志格式：简短，详细，XML，JSON，MagicTree，RubyObject，MongoDB，ElasticSearch，SQL。
代理支持包括TOR
自定义HTTP标头
基本HTTP身份验证
控制网页重定向
IP地址范围
模糊匹配
结果确定性意识
在命令行上定义的自定义插件
IDN（国际域名）支持

WhatWeb安装

WhatWeb是跨平台兼容的，适用于任何Ruby 2.x环境，包括Windows，Mac OSX和Linux。

参考：https://github.com/urbanadventurer/WhatWeb/wiki/Installation

WhatWeb示例用法

使用WhatWeb扫描reddit.com。

$ ./whatweb reddit.com
http://reddit.com [301 Moved Permanently] Country[UNITED STATES][US], HTTPServer[snooserv], IP[151.101.65.140], RedirectLocation[https://www.reddit.com/], UncommonHeaders[retry-after,x-served-by,x-cache-hits,x-timer], Via-Proxy[1.1 varnish]
https://www.reddit.com/ [200 OK] Cookies[edgebucket,eu_cookie_v2,loid,rabt,rseor3,session_tracker,token], Country[UNITED STATES][US], Email[banner@2x.png,snoo-home@2x.png], Frame, HTML5, HTTPServer[snooserv], HttpOnly[token], IP[151.101.37.140], Open-Graph-Protocol[website], Script[text/javascript], Strict-Transport-Security[max-age=15552000; includeSubDomains; preload], Title[reddit: the front page of the internet], UncommonHeaders[fastly-restarts,x-served-by,x-cache-hits,x-timer], Via-Proxy[1.1 varnish], X-Frame-Options[SAMEORIGIN]

$ ./whatweb reddit.com

http://reddit.com [301 Moved Permanently] Country[UNITED STATES][US], HTTPServer[snooserv], IP[151.101.65.140], RedirectLocation[https://www.reddit.com/], UncommonHeaders[retry-after,x-served-by,x-cache-hits,x-timer], Via-Proxy[1.1 varnish]

https://www.reddit.com/ [200 OK] Cookies[edgebucket,eu_cookie_v2,loid,rabt,rseor3,session_tracker,token], Country[UNITED STATES][US], Email[banner@2x.png,snoo-home@2x.png], Frame, HTML5, HTTPServer[snooserv], HttpOnly[token], IP[151.101.37.140], Open-Graph-Protocol[website], Script[text/javascript], Strict-Transport-Security[max-age=15552000; includeSubDomains; preload], Title[reddit: the front page of the internet], UncommonHeaders[fastly-restarts,x-served-by,x-cache-hits,x-timer], Via-Proxy[1.1 varnish], X-Frame-Options[SAMEORIGIN]

参数

Usage: whatweb [options] <URLs>

TARGET SELECTION:
  <TARGETs>             Enter URLs, hostnames, IP addresses, filenames or
                        IP ranges in CIDR, x.x.x-x, or x.x.x.x-x.x.x.x
                        format.
  --input-file=FILE, -i Read targets from a file. You can pipe
                        hostnames or URLs directly with -i /dev/stdin.

TARGET MODIFICATION:
  --url-prefix          Add a prefix to target URLs.
  --url-suffix          Add a suffix to target URLs.
  --url-pattern         Insert the targets into a URL. Requires --input-file,
                        eg. www.example.com/%insert%/robots.txt 

AGGRESSION:
  The aggression level controls the trade-off between speed/stealth and
  reliability.
  --aggression, -a=LEVEL Set the aggression level. Default: 1.
  Aggression levels are:
  1. Stealthy   Makes one HTTP request per target. Also follows redirects.
  3. Aggressive If a level 1 plugin is matched, additional requests will be
      made.
  4. Heavy      Makes a lot of HTTP requests per target. Aggressive tests from
      all plugins are used for all URLs.

HTTP OPTIONS:
  --user-agent, -U=AGENT Identify as AGENT instead of WhatWeb/0.5.0.
  --header, -H          Add an HTTP header. eg "Foo:Bar". Specifying a default
                        header will replace it. Specifying an empty value, eg.
                        "User-Agent:" will remove the header.
  --follow-redirect=WHEN Control when to follow redirects. WHEN may be `never',
                        `http-only', `meta-only', `same-site', or `always'.
                        Default: always.
  --max-redirects=NUM   Maximum number of contiguous redirects. Default: 10.

AUTHENTICATION:
  --user, -u=<user:password> HTTP basic authentication.
  --cookie, -c=COOKIES  Provide cookies, e.g. 'name=value; name2=value2'.
  --cookiejar=FILE      Read cookies from a file.

PROXY:
  --proxy           <hostname[:port]> Set proxy hostname and port.
                    Default: 8080.
  --proxy-user      <username:password> Set proxy user and password.

PLUGINS:
  --list-plugins, -l            List all plugins.
  --info-plugins, -I=[SEARCH]   List all plugins with detailed information.
                                Optionally search with keywords in a comma
                                delimited list.
  --search-plugins=STRING       Search plugins for a keyword.
  --plugins, -p=LIST  Select plugins. LIST is a comma delimited set of 
                      selected plugins. Default is all.
                      Each element can be a directory, file or plugin name and
                      can optionally have a modifier, eg. + or -
                      Examples: +/tmp/moo.rb,+/tmp/foo.rb
                      title,md5,+./plugins-disabled/
                      ./plugins-disabled,-md5
                      -p + is a shortcut for -p +plugins-disabled.

  --grep, -g=STRING|REGEXP      Search for STRING or a Regular Expression. Shows 
                                only the results that match.
                                Examples: --grep "hello"
                                --grep "/he[l]*o/"
  --custom-plugin=DEFINITION\tDefine a custom plugin named Custom-Plugin,
  --custom-plugin=DEFINITION  Define a custom plugin named Custom-Plugin,
                        Examples: ":text=>'powered by abc'"
                        ":version=>/powered[ ]?by ab[0-9]/"
                        ":ghdb=>'intitle:abc \"powered by abc\"'"
                        ":md5=>'8666257030b94d3bdb46e05945f60b42'"
  --dorks=PLUGIN        List Google dorks for the selected plugin.

OUTPUT:
  --verbose, -v         Verbose output includes plugin descriptions. Use twice
                        for debugging.
  --colour,--color=WHEN control whether colour is used. WHEN may be `never',
                        `always', or `auto'.
  --quiet, -q           Do not display brief logging to STDOUT.
  --no-errors           Suppress error messages.

LOGGING:
  --log-brief=FILE        Log brief, one-line output.
  --log-verbose=FILE      Log verbose output.
  --log-errors=FILE       Log errors.
  --log-xml=FILE          Log XML format.
  --log-json=FILE         Log JSON format.
  --log-sql=FILE          Log SQL INSERT statements.
  --log-sql-create=FILE   Create SQL database tables.
  --log-json-verbose=FILE Log JSON Verbose format.
  --log-magictree=FILE    Log MagicTree XML format.
  --log-object=FILE       Log Ruby object inspection format.
  --log-mongo-database    Name of the MongoDB database.
  --log-mongo-collection  Name of the MongoDB collection. Default: whatweb.
  --log-mongo-host        MongoDB hostname or IP address. Default: 0.0.0.0.
  --log-mongo-username    MongoDB username. Default: nil.
  --log-mongo-password    MongoDB password. Default: nil.  
  --log-elastic-index     Name of the index to store results. Default: whatweb 
  --log-elastic-host      Host:port of the elastic http interface. Default: 127.0.0.1:9200
  
PERFORMANCE & STABILITY:
  --max-threads, -t       Number of simultaneous threads. Default: 25.
  --open-timeout          Time in seconds. Default: 15.
  --read-timeout          Time in seconds. Default: 30.
  --wait=SECONDS          Wait SECONDS between connections.
                          This is useful when using a single thread.

HELP & MISCELLANEOUS:
  --short-help            Short usage help.
  --help, -h              Complete usage help.
  --debug                 Raise errors in plugins.
  --version               Display version information. (WhatWeb 0.5.0).

EXAMPLE USAGE:
* Scan example.com.
  ./whatweb example.com
* Scan reddit.com slashdot.org with verbose plugin descriptions.
  ./whatweb -v reddit.com slashdot.org
* An aggressive scan of wired.com detects the exact version of WordPress.
  ./whatweb -a 3 www.wired.com
* Scan the local network quickly and suppress errors.
  whatweb --no-errors 192.168.0.0/24
* Scan the local network for https websites.
  whatweb --no-errors --url-prefix https:// 192.168.0.0/24
* Scan for crossdomain policies in the Alexa Top 1000.
  ./whatweb -i plugin-development/alexa-top-100.txt \
  --url-suffix /crossdomain.xml -p crossdomain_xml

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

Usage: whatweb [options] <URLs>

TARGET SELECTION:

<TARGETs> Enter URLs, hostnames, IP addresses, filenames or

IP ranges in CIDR, x.x.x-x, or x.x.x.x-x.x.x.x

format.

--input-file=FILE, -i Read targets from a file. You can pipe

hostnames or URLs directly with -i /dev/stdin.

TARGET MODIFICATION:

--url-prefix Add a prefix to target URLs.

--url-suffix Add a suffix to target URLs.

--url-pattern Insert the targets into a URL. Requires --input-file,

eg. www.example.com/%insert%/robots.txt

AGGRESSION:

The aggression level controls the trade-off between speed/stealth and

reliability.

--aggression, -a=LEVEL Set the aggression level. Default: 1.

Aggression levels are:

1. Stealthy Makes one HTTP request per target. Also follows redirects.

3. Aggressive If a level 1 plugin is matched, additional requests will be

made.

4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from

all plugins are used for all URLs.

HTTP OPTIONS:

--user-agent, -U=AGENT Identify as AGENT instead of WhatWeb/0.5.0.

--header, -H Add an HTTP header. eg "Foo:Bar". Specifying a default

header will replace it. Specifying an empty value, eg.

"User-Agent:" will remove the header.

--follow-redirect=WHEN Control when to follow redirects. WHEN may be `never',

`http-only', `meta-only', `same-site', or `always'.

Default: always.

--max-redirects=NUM Maximum number of contiguous redirects. Default: 10.

AUTHENTICATION:

--user, -u=<user:password> HTTP basic authentication.

--cookie, -c=COOKIES Provide cookies, e.g. 'name=value; name2=value2'.

--cookiejar=FILE Read cookies from a file.

PROXY:

--proxy <hostname[:port]> Set proxy hostname and port.

Default: 8080.

--proxy-user <username:password> Set proxy user and password.

PLUGINS:

--list-plugins, -l List all plugins.

--info-plugins, -I=[SEARCH] List all plugins with detailed information.

Optionally search with keywords in a comma

delimited list.

--search-plugins=STRING Search plugins for a keyword.

--plugins, -p=LIST Select plugins. LIST is a comma delimited set of

selected plugins. Default is all.

Each element can be a directory, file or plugin name and

can optionally have a modifier, eg. + or -

Examples: +/tmp/moo.rb,+/tmp/foo.rb

title,md5,+./plugins-disabled/

./plugins-disabled,-md5

-p + is a shortcut for -p +plugins-disabled.

--grep, -g=STRING|REGEXP Search for STRING or a Regular Expression. Shows

only the results that match.

Examples: --grep "hello"

--grep "/he[l]*o/"

--custom-plugin=DEFINITION\tDefine a custom plugin named Custom-Plugin,

--custom-plugin=DEFINITION Define a custom plugin named Custom-Plugin,

Examples: ":text=>'powered by abc'"

":version=>/powered[ ]?by ab[0-9]/"

":ghdb=>'intitle:abc \"powered by abc\"'"

":md5=>'8666257030b94d3bdb46e05945f60b42'"

--dorks=PLUGIN List Google dorks for the selected plugin.

OUTPUT:

--verbose, -v Verbose output includes plugin descriptions. Use twice

for debugging.

--colour,--color=WHEN control whether colour is used. WHEN may be `never',

`always', or `auto'.

--quiet, -q Do not display brief logging to STDOUT.

--no-errors Suppress error messages.

LOGGING:

--log-brief=FILE Log brief, one-line output.

--log-verbose=FILE Log verbose output.

--log-errors=FILE Log errors.

--log-xml=FILE Log XML format.

--log-json=FILE Log JSON format.

--log-sql=FILE Log SQL INSERT statements.

--log-sql-create=FILE Create SQL database tables.

--log-json-verbose=FILE Log JSON Verbose format.

--log-magictree=FILE Log MagicTree XML format.

--log-object=FILE Log Ruby object inspection format.

--log-mongo-database Name of the MongoDB database.

--log-mongo-collection Name of the MongoDB collection. Default: whatweb.

--log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0.

--log-mongo-username MongoDB username. Default: nil.

--log-mongo-password MongoDB password. Default: nil.

--log-elastic-index Name of the index to store results. Default: whatweb

--log-elastic-host Host:port of the elastic http interface. Default: 127.0.0.1:9200

PERFORMANCE & STABILITY:

--max-threads, -t Number of simultaneous threads. Default: 25.

--open-timeout Time in seconds. Default: 15.

--read-timeout Time in seconds. Default: 30.

--wait=SECONDS Wait SECONDS between connections.

This is useful when using a single thread.

HELP & MISCELLANEOUS:

--short-help Short usage help.

--help, -h Complete usage help.

--debug Raise errors in plugins.

--version Display version information. (WhatWeb 0.5.0).

EXAMPLE USAGE:

* Scan example.com.

./whatweb example.com

* Scan reddit.com slashdot.org with verbose plugin descriptions.

./whatweb -v reddit.com slashdot.org

* An aggressive scan of wired.com detects the exact version of WordPress.

./whatweb -a 3 www.wired.com

* Scan the local network quickly and suppress errors.

whatweb --no-errors 192.168.0.0/24

* Scan the local network for https websites.

whatweb --no-errors --url-prefix https:// 192.168.0.0/24

* Scan for crossdomain policies in the Alexa Top 1000.

./whatweb -i plugin-development/alexa-top-100.txt \

--url-suffix /crossdomain.xml -p crossdomain_xml

记录和输出

支持以下类型的日志记录：

--log-brief = FILE Brief，one-line，greppable format
--log-verbose = FILE详细
--log-xml = FILE XML格式。提供了XSL样式表
--log-json = FILE JSON格式
--log-json-verbose = FILE JSON详细格式
--log-magictree = FILE MagicTree XML格式
--log-object = FILE Ruby对象检查格式
--log-mongo-database MongoDB数据库的名称
--log-mongo-collection MongoDB集合的名称。默认值：whatweb
--log-mongo-host MongoDB主机名或IP地址。默认值：0.0.0.0
--log-mongo-username MongoDB用户名。默认值：nil
--log-mongo-password MongoDB密码。默认值：nil
--log-elastic-index用于存储结果的索引的名称。默认值：whatweb
--log-elastic-host Host：弹性http接口的端口。默认值：127.0.0.1：9200
--log-errors = FILE记录错误。这通常以红色打印到屏幕上。

您可以通过指定多个命令行日志记录选项同时输出到多个日志。想要SQL输出的高级用户应阅读源代码以查看不支持的功能。

WhatWeb插件

比赛用：

文本字符串（区分大小写）
常用表达
Google Hack数据库查询（有限的关键字集）
MD5哈希
URL识别
HTML标记模式
用于被动和激进操作的自定义ruby代码

列出支持的插件：

$ ./whatweb -l

1	$ ./whatweb -l

搜索插件

$ ./whatweb -I phpBB

WhatWeb Detailed Plugin List
Searching for phpBB
================================================================================
Plugin:         phpBB
--------------------------------------------------------------------------------
Description:    phpBB is a free forum 
Website:        http://phpbb.org/

Author:         Andrew Horton
Version:        0.3

Features:       [Yes]  Pattern Matching (7)
                [Yes]  Version detection from pattern matching
                [Yes]  Function for passive matches
                [Yes]  Function for aggressive matches
                [Yes]  Google Dorks (1)

Google Dorks:
[1] "Powered by phpBB"
================================================================================