FOSS Security Tools: WhatWeb

In this second article in the series on FOSS security tools, we will learn about WhatWeb, a next generation web scanner.

WhatWeb can be used to recognise web servers, JavaScript libraries, blogging platforms and content management systems (CMS). It is written by Andrew Horton and Brendan Coles using the Ruby programming language. The source code is released under the GNU General Public License v2.0. You can install the same on Ubuntu using the following command:

$ sudo apt install whatweb

The –version argument can be used to identify the installed version as shown below:

$ whatweb --version
WhatWeb version 0.5.5 ( https://www.morningstarsecurity.com/research/whatweb/ )

Help

You can view the help summary using the -h option as indicated below:

$ whatweb -h
...

WhatWeb - Next generation web scanner version 0.5.5.
Developed by Andrew Horton (urbanadventurer) and Brendan Coles (bcoles).
Homepage: https://www.morningstarsecurity.com/research/whatweb
Usage: whatweb [options] <URLs>
TARGET SELECTION:

<TARGETs> Enter URLs, hostnames, IP addresses, filenames or
IP ranges in CIDR, x.x.x-x, or x.x.x.x-x.x.x.x
format.
--input-file=FILE, -i Read targets from a file. You can pipe
hostnames or URLs directly with -i /dev/stdin.
TARGET MODIFICATION:
--url-prefix Add a prefix to target URLs.
--url-suffix Add a suffix to target URLs.
--url-pattern Insert the targets into a URL.
e.g. example.com/%insert%/robots.txt

Verbosity

The verbosity level can be specified with the -v argument as follows:

$ whatweb -v google.com
WhatWeb report for http://google.com
Status : 301 Moved Permanently
Title : 301 Moved
IP : 142.250.196.46
Country : UNITED STATES, US
Summary : HTTPServer[gws], RedirectLocation[http://www.google.com/], UncommonHeaders[content-security-policy-report-only], X-Frame-Options[SAMEORIGIN], X-XSS-Protection[0]

You can also increase the verbosity of the output by using -vv with the command.

Plugins

WhatWeb supports over 1800 plugins. You can view the list of supported plugins using the -l option as shown below:

$ whatweb -l
WhatWeb Plugin List
Plugin Website
-------------------------------------------------------------

1&1 Hosting https://www.1and1.com/
1024-CMS http://www.1024cms.org/

360-Web-Manager http://www.360webmanager.com

3COM-NBX http://inpath.com/products/3com-nbx.html

3dcart http://www.3dcart.com/

4D http://www.4d.com/products.html

4images http://www.4homepages.de/

68-Classifieds-Script http://www.68classifieds.com/

6kbbs http://www.6kbbs.net/

Aardvark-Topsites-PHP http://www.aardvarktopsitesphp.com/topsites/

AB-WEB-CMS http://www.aeline-informatique.com/

ABO_CMS http://www.abocms.com/

...

The output report provides a list of the detected plugins and their observed values:

Detected Plugins:
[ HTTPServer ]
	HTTP server header string. This plugin also attempts to 
	identify the operating system from the server header. 

	String       : gws (from server string)

[ RedirectLocation ]
	HTTP Server string location. used with http-status 301 and 
	302 

	String       : http://www.google.com/ (from location)

[ UncommonHeaders ]
	Uncommon HTTP server headers. The blacklist includes all 
	the standard headers and many non standard but common ones. 
	Interesting but fairly common headers should have their own 
	plugins, eg. x-powered-by, server and x-aspnet-version. 
	Info about headers can be found at www.http-stats.com 

	String       : content-security-policy-report-only (from headers)

HTTP headers

The HTTP header options and values are also provided in the report:

[ X-Frame-Options ]
	This plugin retrieves the X-Frame-Options value from the 
	HTTP header. - More Info: 
	http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
	aspx

	String       : SAMEORIGIN

[ X-XSS-Protection ]
	This plugin retrieves the X-XSS-Protection value from the 
	HTTP header. - More Info: 
	http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
	aspx

	String       : 0

HTTP Headers:
	HTTP/1.1 301 Moved Permanently
	Location: http://www.google.com/
	Content-Type: text/html; charset=UTF-8
	Content-Security-Policy-Report-Only: object-src ‘none’;base-uri ‘self’;script-src ‘nonce-XPTH2L46zvJo-axPRbsFhw’ ‘strict-dynamic’ ‘report-sample’ ‘unsafe-eval’ ‘unsafe-inline’ https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
	Date: Mon, 06 May 2024 17:14:23 GMT
	Expires: Wed, 05 Jun 2024 17:14:23 GMT
	Cache-Control: public, max-age=2592000
	Server: gws
	Content-Length: 219
	X-XSS-Protection: 0
	X-Frame-Options: SAMEORIGIN
	Connection: close

Report

A more detailed report follows the header information as illustrated below:

WhatWeb report for http://www.google.com/
Status    : 200 OK
Title     : Google
IP        : 142.250.205.228
Country   : UNITED STATES, US

Summary   : Cookies[AEC,NID], HTML5, HTTPServer[gws], HttpOnly[AEC,NID], Script, UncommonHeaders[content-security-policy-report-only], X-Frame-Options[SAMEORIGIN], X-XSS-Protection[0]

Detected Plugins:
[ Cookies ]
	Display the names of cookies in the HTTP headers. The 
	values are not returned to save on space. 

	String       : AEC
	String       : NID

Information on the HTML version, operating system of the server, HTTP header, and cross-site scripting (XSS) is also detected.

[ HTML5 ]
	HTML version 5, detected by the doctype declaration 

[ HTTPServer ]
	HTTP server header string. This plugin also attempts to 
	identify the operating system from the server header. 

	String       : gws (from server string)

[ HttpOnly ]
	If the HttpOnly flag is included in the HTTP set-cookie 
	response header and the browser supports it then the cookie 
	cannot be accessed through client side script - More Info: 
	http://en.wikipedia.org/wiki/HTTP_cookie 

	String       : AEC,NID

[ Script ]
	This plugin detects instances of script HTML elements and 
	returns the script language/type. 


[ UncommonHeaders ]
	Uncommon HTTP server headers. The blacklist includes all 
	the standard headers and many non standard but common ones. 
	Interesting but fairly common headers should have their own 
	plugins, eg. x-powered-by, server and x-aspnet-version. 
	Info about headers can be found at www.http-stats.com 

	String       : content-security-policy-report-only (from headers)

[ X-Frame-Options ]
	This plugin retrieves the X-Frame-Options value from the 
	HTTP header. - More Info: 
	http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
	aspx

	String       : SAMEORIGIN

[ X-XSS-Protection ]
	This plugin retrieves the X-XSS-Protection value from the 
	HTTP header. - More Info: 
	http://msdn.microsoft.com/en-us/library/cc288472%28VS.85%29.
	aspx

	String       : 0

HTTP Headers:
	HTTP/1.1 200 OK
	Date: Mon, 06 May 2024 17:14:24 GMT
	Expires: -1
	Cache-Control: private, max-age=0
	Content-Type: text/html; charset=ISO-8859-1
	Content-Security-Policy-Report-Only: object-src ‘none’;base-uri ‘self’;script-src ‘nonce-C9_2RdD0iRWxcpf9bL1mhQ’ ‘strict-dynamic’ ‘report-sample’ ‘unsafe-eval’ ‘unsafe-inline’ https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
	P3P: CP=”This is not a P3P policy! See g.co/p3phelp for more info.”
	Content-Encoding: gzip
	Server: gws
	Content-Length: 8849
	X-XSS-Protection: 0
	X-Frame-Options: SAMEORIGIN
	Set-Cookie: AEC=AQTF6HzW1KsaRuWA27yzb8WuUTE9eGyz1mSvPYELtTvc2yZWsfBx29nLg54; expires=Sat, 02-Nov-2024 17:14:24 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
	Set-Cookie: NID=513=nz62flmI6wajdtIip_abZGOKZ86o3mb2qHZS CeN2BivtYZelBDWsooM_JjAHgWzcE21fuNQyfppX_KDu3IWC8WbZWlDXMEXKCjA9wbcmP8fbupmzD8p1t5P6guDdVuJbb8rfOhUozLt 4yrd9rIVTNEedx6ghqCQlqt7SZuvqtBA; expires=Tue, 05-Nov-2024 17:14:24 GMT; path=/; domain=.google.com; HttpOnly
	Connection: close

URLs

You can provide a list of URLs to be scanned from a file using the –input-file option as shown below:

$ whatweb --input-file urls.txt

Logging

You can also log the output to a file and other external entities rather than the console. The supported logging outputs are given below:

Embedded devices

The WhatWeb command can also be run on embedded devices. The following is an example from scanning a TP-Link router device:

$ whatweb -v 10.0.0.5
WhatWeb report for http://10.0.0.5
Status    : 200 OK
Title     : <None>
IP        : 10.0.0.5
Country   : RESERVED, ZZ

Summary   : Cookies[JSESSIONID], Email[zhujunjie@tp-link.net], Frame, HttpOnly[JSESSIONID], JQuery[1.8.3], PasswordField, Script[text/javascript]

Detected Plugins:
[ Cookies ]
	Display the names of cookies in the HTTP headers. The 
	values are not returned to save on space. 

	String       : JSESSIONID

The router has an email contact for reference, and hence the same is provided in the output:

[ Email ]
	Extract email addresses. Find valid email address and 
	syntactically invalid email addresses from mailto: link 
	tags. We match syntactically invalid links containing 
	mailto: to catch anti-spam email addresses, eg. bob at 
	gmail.com. This uses the simplified email regular 
	expression from 
	http://www.regular-expressions.info/email.html for valid 
	email address matching. 

	String       : zhujunjie@tp-link.net

iframes, JavaScript libraries, and HTTP headers are also detected as illustrated below:

[ Frame ]
	This plugin detects instances of frame and iframe HTML 
	elements. 


[ HttpOnly ]
	If the HttpOnly flag is included in the HTTP set-cookie 
	response header and the browser supports it then the cookie 
	cannot be accessed through client side script - More Info: 
	http://en.wikipedia.org/wiki/HTTP_cookie 

	String       : JSESSIONID

[ JQuery ]
	A fast, concise, JavaScript that simplifies how to traverse 
	HTML documents, handle events, perform animations, and add 
	AJAX. 

	Version      : 1.8.3
	Website     : http://jquery.com/

[ PasswordField ]
	find password fields 

[ Script ]
	This plugin detects instances of script HTML elements and 
	returns the script language/type. 

	String       : text/javascript

HTTP Headers:
	HTTP/1.1 200 OK
	Content-Type: text/html; charset=utf-8
	Content-Length: 62924
	Set-Cookie: JSESSIONID=deleted; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; HttpOnly
	Connection: close

Scan modes

The different scan levels are given below:

The aggressive mode level can be set to the highest value of 3, and along with the increased verbosity level you can get a more detailed scan output. For example:

$ whatweb -vv -a 3 google.com
http://google.com/ [301]
redirect 1 from http://google.com to http://www.google.com/
Identifying: http://google.com
HTTP-Status: 301
[[“Country”, [{:string=>”UNITED STATES”, :module=>”US”, :certainty=>100}]],
 [“HTTPServer”, [{:name=>”server string”, :string=>”gws”, :certainty=>100}]],
 [“IP”, [{:string=>”142.250.196.46”, :certainty=>100}]],
 [“RedirectLocation”,
  [{:name=>”location”, :string=>”http://www.google.com/”, :certainty=>100}]],
 [“Title”, [{:name=>”page title”, :string=>”301 Moved”, :certainty=>100}]],
 [“UncommonHeaders”,
  [{:name=>”headers”,
    :string=>”content-security-policy-report-only”,
    :certainty=>100}]],
 [“X-Frame-Options”, [{:string=>”SAMEORIGIN”, :certainty=>100}]],
 [“X-XSS-Protection”, [{:string=>”0”, :certainty=>100}]]]

WhatWeb report for http://google.com
Status    : 301 Moved Permanently
Title     : 301 Moved
IP        : 142.250.196.46
Country   : UNITED STATES, US

Summary   : HTTPServer[gws], RedirectLocation[http://www.google.com/], UncommonHeaders[content-security-policy-report-only], X-Frame-Options[SAMEORIGIN], X-XSS-Protection[0]
...
Detected Plugins:
[ Cookies ]
	Display the names of cookies in the HTTP headers. The 
	values are not returned to save on space. 

	String       : 1P_JAR
	{:certainty=>100, :string=>”1P_JAR”}
	String       : AEC
	{:certainty=>100, :string=>”AEC”}
	String       : NID
	{:certainty=>100, :string=>”NID”}
...

Performance

WhatWeb also comes with several options to increase the performance of the scan:

You are encouraged to read the manual page of WhatWeb to learn more on its usage, plugins, and command arguments.