FOSS Security Tools: WhatWeb

By
Shakthi Kannan
-
0
238
FOSS Security Tools: WhatWeb

In this second article in the series on FOSS security tools, we will learn about WhatWeb, a next generation web scanner.

WhatWeb can be used to recognise web servers, JavaScript libraries, blogging platforms and content management systems (CMS). It is written by Andrew Horton and Brendan Coles using the Ruby programming language. The source code is released under the GNU General Public License v2.0. You can install the same on Ubuntu using the following command:

$ sudo apt install whatweb

The –version argument can be used to identify the installed version as shown below:

$ whatweb --version
WhatWeb version 0.5.5 ( https://www.morningstarsecurity.com/research/whatweb/ )

Help

You can view the help summary using the -h option as indicated below:

$ whatweb -h
...
 
WhatWeb - Next generation web scanner version 0.5.5.
Developed by Andrew Horton (urbanadventurer) and Brendan Coles (bcoles).
Homepage: https://www.morningstarsecurity.com/research/whatweb
Usage: whatweb [options] <URLs>
TARGET SELECTION:
 
<TARGETs> Enter URLs, hostnames, IP addresses, filenames or
IP ranges in CIDR, x.x.x-x, or x.x.x.x-x.x.x.x
format.
--input-file=FILE, -i Read targets from a file. You can pipe
hostnames or URLs directly with -i /dev/stdin.
TARGET MODIFICATION:
--url-prefix Add a prefix to target URLs.
--url-suffix Add a suffix to target URLs.
--url-pattern Insert the targets into a URL.
e.g. example.com/%insert%/robots.txt

Verbosity

The verbosity level can be specified with the -v argument as follows:

$ whatweb -v google.com
WhatWeb report for http://google.com
Status : 301 Moved Permanently
Title : 301 Moved
IP : 142.250.196.46
Country : UNITED STATES, US
Summary : HTTPServer[gws], RedirectLocation[http://www.google.com/], UncommonHeaders[content-security-policy-report-only], X-Frame-Options[SAMEORIGIN], X-XSS-Protection[0]

You can also increase the verbosity of the output by using -vv with the command.

Plugins

WhatWeb supports over 1800 plugins. You can view the list of supported plugins using the -l option as shown below:

$ whatweb -l
WhatWeb Plugin List
Plugin Website
-------------------------------------------------------------
 
1&1 Hosting https://www.1and1.com/
1024-CMS http://www.1024cms.org/
 
360-Web-Manager http://www.360webmanager.com
 
3COM-NBX http://inpath.com/products/3com-nbx.html
 
3dcart http://www.3dcart.com/
 
4D http://www.4d.com/products.html
 
4images http://www.4homepages.de/
 
68-Classifieds-Script http://www.68classifieds.com/
 
6kbbs http://www.6kbbs.net/
 
Aardvark-Topsites-PHP http://www.aardvarktopsitesphp.com/topsites/
 
AB-WEB-CMS http://www.aeline-informatique.com/
 
ABO_CMS http://www.abocms.com/
 
...

The output report provides a list of the detected plugins and their observed values:

Detected Plugins:
[ HTTPServer ]
    HTTP server header string. This plugin also attempts to
    identify the operating system from the server header.
 
    String       : gws (from server string)
 
[ RedirectLocation ]
    HTTP Server string location. used with http-status 301 and
    302
 
    String       : http://www.google.com/ (from location)
 
[ UncommonHeaders ]
    Uncommon HTTP server headers. The blacklist includes all
    the standard headers and many non standard but common ones.
    Interesting but fairly common headers should have their own
    plugins, eg. x-powered-by, server and x-aspnet-version.
    Info about headers can be found at www.http-stats.com
 
    String       : content-security-policy-report-only (from headers)

HTTP headers

The HTTP header options and values are also provided in the report:

[ X-Frame-Options ]
    This plugin retrieves the X-Frame-Options value from the
    HTTP header. - More Info:
    http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
    aspx
 
    String       : SAMEORIGIN
 
[ X-XSS-Protection ]
    This plugin retrieves the X-XSS-Protection value from the
    HTTP header. - More Info:
    http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
    aspx
 
    String       : 0
 
HTTP Headers:
    HTTP/1.1 301 Moved Permanently
    Location: http://www.google.com/
    Content-Type: text/html; charset=UTF-8
    Content-Security-Policy-Report-Only: object-src ‘none’;base-uri ‘self’;script-src ‘nonce-XPTH2L46zvJo-axPRbsFhw’ ‘strict-dynamic’ ‘report-sample’ ‘unsafe-eval’ ‘unsafe-inline’ https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    Date: Mon, 06 May 2024 17:14:23 GMT
    Expires: Wed, 05 Jun 2024 17:14:23 GMT
    Cache-Control: public, max-age=2592000
    Server: gws
    Content-Length: 219
    X-XSS-Protection: 0
    X-Frame-Options: SAMEORIGIN
    Connection: close

Report

A more detailed report follows the header information as illustrated below:

WhatWeb report for http://www.google.com/
Status    : 200 OK
Title     : Google
IP        : 142.250.205.228
Country   : UNITED STATES, US
 
Summary   : Cookies[AEC,NID], HTML5, HTTPServer[gws], HttpOnly[AEC,NID], Script, UncommonHeaders[content-security-policy-report-only], X-Frame-Options[SAMEORIGIN], X-XSS-Protection[0]
 
Detected Plugins:
[ Cookies ]
    Display the names of cookies in the HTTP headers. The
    values are not returned to save on space.
 
    String       : AEC
    String       : NID

Information on the HTML version, operating system of the server, HTTP header, and cross-site scripting (XSS) is also detected.

[ HTML5 ]
    HTML version 5, detected by the doctype declaration
 
[ HTTPServer ]
    HTTP server header string. This plugin also attempts to
    identify the operating system from the server header.
 
    String       : gws (from server string)
 
[ HttpOnly ]
    If the HttpOnly flag is included in the HTTP set-cookie
    response header and the browser supports it then the cookie
    cannot be accessed through client side script - More Info:
    http://en.wikipedia.org/wiki/HTTP_cookie
 
    String       : AEC,NID
 
[ Script ]
    This plugin detects instances of script HTML elements and
    returns the script language/type.
 
 
[ UncommonHeaders ]
    Uncommon HTTP server headers. The blacklist includes all
    the standard headers and many non standard but common ones.
    Interesting but fairly common headers should have their own
    plugins, eg. x-powered-by, server and x-aspnet-version.
    Info about headers can be found at www.http-stats.com
 
    String       : content-security-policy-report-only (from headers)
 
[ X-Frame-Options ]
    This plugin retrieves the X-Frame-Options value from the
    HTTP header. - More Info:
    http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
    aspx
 
    String       : SAMEORIGIN
 
[ X-XSS-Protection ]
    This plugin retrieves the X-XSS-Protection value from the
    HTTP header. - More Info:
    http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
    aspx
 
    String       : 0
 
HTTP Headers:
    HTTP/1.1 200 OK
    Date: Mon, 06 May 2024 17:14:24 GMT
    Expires: -1
    Cache-Control: private, max-age=0
    Content-Type: text/html; charset=ISO-8859-1
    Content-Security-Policy-Report-Only: object-src ‘none’;base-uri ‘self’;script-src ‘nonce-C9_2RdD0iRWxcpf9bL1mhQ’ ‘strict-dynamic’ ‘report-sample’ ‘unsafe-eval’ ‘unsafe-inline’ https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    P3P: CP=”This is not a P3P policy! See g.co/p3phelp for more info.”
    Content-Encoding: gzip
    Server: gws
    Content-Length: 8849
    X-XSS-Protection: 0
    X-Frame-Options: SAMEORIGIN
    Set-Cookie: AEC=AQTF6HzW1KsaRuWA27yzb8WuUTE9eGyz1mSvPYELtTvc2yZWsfBx29nLg54; expires=Sat, 02-Nov-2024 17:14:24 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
    Set-Cookie: NID=513=nz62flmI6wajdtIip_abZGOKZ86o3mb2qHZS CeN2BivtYZelBDWsooM_JjAHgWzcE21fuNQyfppX_KDu3IWC8WbZWlDXMEXKCjA9wbcmP8fbupmzD8p1t5P6guDdVuJbb8rfOhUozLt 4yrd9rIVTNEedx6ghqCQlqt7SZuvqtBA; expires=Tue, 05-Nov-2024 17:14:24 GMT; path=/; domain=.google.com; HttpOnly
    Connection: close

URLs

You can provide a list of URLs to be scanned from a file using the –input-file option as shown below:

$ whatweb --input-file urls.txt

Logging

You can also log the output to a file and other external entities rather than the console. The supported logging outputs are given below:

Option Entity
–log-json JSON
–log-sql SQL INSERT commands
–log-sql-create Create SQL database tables
–log-xml XML
–log-magictree MagicTree XML
–log-object Ruby object inspection format

–log-mongo-database

MongoDB database

–log-mongo-collection

MongoDB colection

–log-elastic-index

Elasticsearch index

Embedded devices

The WhatWeb command can also be run on embedded devices. The following is an example from scanning a TP-Link router device:

$ whatweb -v 10.0.0.5
WhatWeb report for http://10.0.0.5
Status    : 200 OK
Title     : <None>
IP        : 10.0.0.5
Country   : RESERVED, ZZ
 
Summary   : Cookies[JSESSIONID], Email[zhujunjie@tp-link.net], Frame, HttpOnly[JSESSIONID], JQuery[1.8.3], PasswordField, Script[text/javascript]
 
Detected Plugins:
[ Cookies ]
    Display the names of cookies in the HTTP headers. The
    values are not returned to save on space.
 
    String       : JSESSIONID

The router has an email contact for reference, and hence the same is provided in the output:

[ Email ]
    Extract email addresses. Find valid email address and
    syntactically invalid email addresses from mailto: link
    tags. We match syntactically invalid links containing
    mailto: to catch anti-spam email addresses, eg. bob at
    gmail.com. This uses the simplified email regular
    expression from
    http://www.regular-expressions.info/email.html for valid
    email address matching.
 
    String       : zhujunjie@tp-link.net

iframes, JavaScript libraries, and HTTP headers are also detected as illustrated below:

[ Frame ]
    This plugin detects instances of frame and iframe HTML
    elements.
 
 
[ HttpOnly ]
    If the HttpOnly flag is included in the HTTP set-cookie
    response header and the browser supports it then the cookie
    cannot be accessed through client side script - More Info:
    http://en.wikipedia.org/wiki/HTTP_cookie
 
    String       : JSESSIONID
 
[ JQuery ]
    A fast, concise, JavaScript that simplifies how to traverse
    HTML documents, handle events, perform animations, and add
    AJAX.
 
    Version      : 1.8.3
    Website     : http://jquery.com/
 
[ PasswordField ]
    find password fields
 
[ Script ]
    This plugin detects instances of script HTML elements and
    returns the script language/type.
 
    String       : text/javascript
 
HTTP Headers:
    HTTP/1.1 200 OK
    Content-Type: text/html; charset=utf-8
    Content-Length: 62924
    Set-Cookie: JSESSIONID=deleted; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; HttpOnly
    Connection: close

Scan modes

The different scan levels are given below:

Level Aggression
1 Passive
2 Polite (not implemented)
3 Impolite
4 Aggressive

The aggressive mode level can be set to the highest value of 3, and along with the increased verbosity level you can get a more detailed scan output. For example:

$ whatweb -vv -a 3 google.com
http://google.com/ [301]
redirect 1 from http://google.com to http://www.google.com/
Identifying: http://google.com
HTTP-Status: 301
[[“Country”, [{:string=>”UNITED STATES”, :module=>”US”, :certainty=>100}]],
 [“HTTPServer”, [{:name=>”server string”, :string=>”gws”, :certainty=>100}]],
 [“IP”, [{:string=>”142.250.196.46”, :certainty=>100}]],
 [“RedirectLocation”,
  [{:name=>”location”, :string=>”http://www.google.com/”, :certainty=>100}]],
 [“Title”, [{:name=>”page title”, :string=>”301 Moved”, :certainty=>100}]],
 [“UncommonHeaders”,
  [{:name=>”headers”,
    :string=>”content-security-policy-report-only”,
    :certainty=>100}]],
 [“X-Frame-Options”, [{:string=>”SAMEORIGIN”, :certainty=>100}]],
 [“X-XSS-Protection”, [{:string=>”0”, :certainty=>100}]]]
 
WhatWeb report for http://google.com
Status    : 301 Moved Permanently
Title     : 301 Moved
IP        : 142.250.196.46
Country   : UNITED STATES, US
 
Summary   : HTTPServer[gws], RedirectLocation[http://www.google.com/], UncommonHeaders[content-security-policy-report-only], X-Frame-Options[SAMEORIGIN], X-XSS-Protection[0]
...
Detected Plugins:
[ Cookies ]
    Display the names of cookies in the HTTP headers. The
    values are not returned to save on space.
 
    String       : 1P_JAR
    {:certainty=>100, :string=>”1P_JAR”}
    String       : AEC
    {:certainty=>100, :string=>”AEC”}
    String       : NID
    {:certainty=>100, :string=>”NID”}
...

Performance

WhatWeb also comes with several options to increase the performance of the scan:

Option Description
–max-threads Number of simultaneous reads
–read-timeout Time in seconds (Default: 30s)
–open-timout Time in seconds (Default: 15s)
–wait Seconds to wait between connections

You are encouraged to read the manual page of WhatWeb to learn more on its usage, plugins, and command arguments.

RELATED ARTICLESMORE FROM AUTHOR