HCSW and Nmap

HCSW and Nmap

I am an active developer with the excellent open-source Nmap security scanner. I was a fairly infrequent contributor before I participated in Google's Summer of Code. I also write about various aspects of Nmap development in my technical blog.

I got involved with Nmap as soon as I started experimenting with networking using linux and my preferred BSD distribution (at the time NetBSD). I looked around for some interesting network exploration tools, tried many of them out, and discovered that Nmap is by far the most detailed, powerful, fast, and accurate network security scanner available for unix (and probably all other) platforms. I became an Nmap developer because

If you have any comments or questions related to this please send them either to Doug Hoyte (me) or the nmap-dev mailing list.



o Integrated all of your Q32006 service fingerprint submissions.  The
  nmap-service-probe DB grew from 3,671 signatures representing 415
  service protocols to 3,877 signatures representing 426 services.  Big
  thanks to version detection czar Doug Hoyte for doing this.  Notable
  changes are described at http://hcsw.org/blog.pl?a=20&b=20 .

I discovered and fixed a very obscure bug in Nmap's --badsum option. Links to the mailing list thread are posted in this blog entry.

I designed and implemented an experimental new Nmap scan, Qscan. Qscan tries to detect round-trip time discrepancies which might be an indication of packet-forwarding "firewalls" or NAT devices. Transmitting packets over the network takes time. No matter how you structure your networks, network latency is a factor. Transmitting a packet over ethernet, ISDN, wifi, pigeon, etc, inevitably results in a statistically detectable packet transmission delay. This delay is what the Qscan tries to detect and interpret.

Here is the documentation and patch against Nmap 4.52.


o Integrated all 2nd quarter service detection fingerprint
  submissions.  Please keep them coming!  We now have 3,671 signatures
  representing 415 protocols.   Thanks to version detection czar Doug
  Hoyte for doing this.

o Nmap now uses the (relatively) new libpcap pcap_get_selectable_fd
  API on systems which support it.  This means that we no longer need
  to hack the included Pcap to better support Linux.  So Nmap will now
  link with an existing system libpcap by default on that platform if
  one is detected.  Thanks to Doug Hoyte for the patch.

o Fixed a bug which would occasionally cause Nmap to crash with the
  message "log_vwrite: write buffer not large enough".  I thought I
  conquered it in a previous release -- thanks to Doug Hoyte for finding a
  corner case which proved me wrong.

o Fixed a bug in the rDNS system which prevented us from querying
  certain authoritative DNS servers which have recursion explicitly
  disabled.  Thanks to Doug Hoyte for the patch.

o Cleaned up Nmap DNS reporting to be a little more useful and
  concise.  Thanks to Doug Hoyte for the patch.

Nmap 4.20ALPHA4

o Fixed a bug related to bogus completion time estimates when you
  request an estimate (through runtime interaction) right when Nmap is
  starting a subsystem (such as a port scan or version detection).
  Thanks to Diman Todorov for reporting the problem and Doug Hoyte for
  writing a fix.
Nmap 4.11

o Added a dozens of more detailed SSH version detection signatures, thanks
  to a SSH huge survey and integration effort by Doug Hoyte.  The
  results of his large-scale SSH scan are posted at
  http://seclists.org/nmap-dev/2006/Apr-Jun/0393.html .
Nmap 4.10

o Fixed a bug in service detection which could lead to a crash when
  "--version-intensity 0" was used with a UDP scan.  Thanks to Makoto
  Shiotsuki (shio(a)st.rim.or.jp) for reporting the problem and Doug
  Hoyte for producing a patch.

o Changed mass_dns system to print a warning if it can't find any
  available DNS servers, but not quit like it used to.  Thanks to Doug
  Hoyte for the patch.

These are the Nmap CHANGELOG file entries attributed to me before Google's great summer of code. 4.04 is hoped to be a fairly stable release before everybody starts hacking this summer!

Nmap 4.04

o Integrated all of your submissions (about a thousand) from the first
  quarter of this year!  Please keep 'em coming!  The DB has increased
  from 3,153 signatures representing 381 protocols in 4.03 to 3,441
  signatures representing 401 protocols.  No other tool comes close!
  Many of the already existing match lines were improved too.  Thanks
  to Version Detection Czar Doug Hoyte for doing this.

o Fixed a bug which prevented certain TCP+UDP scan commands, such as
  "nmap -sSU -p1-65535 localhost" from scanning both TCP and UDP.
  Instead they gave the error message "WARNING: UDP scan was requested,
  but no udp ports were specified.  Skipping this scan type".  Thanks to
  Doug Hoyte for the patch.

o Nmap has traditionally required you to specify -T* timing options
  before any more granular options like --max-rtt-timeout, otherwise the
  general timing option would overwrite the value from your more
  specific request.  This has now been fixed so that the more specific
  options always have precendence.  Thanks to Doug Hoyte for this patch.

o Nmap now prints a warning when you specify a target name which
  resolves to multiple IP addresses.  Nmap proceeds to scan only the
  first of those addresses (as it always has done).  Thanks to Doug
  Hoyte for the patch.  The warning looks like this:
  Warning: Hostname google.com resolves to 3 IPs. Using

o When debugging (-d) is specified, Nmap now prints a report on the
  timing variables in use.  Thanks to Doug Hoyte for the patch.  The
  report looks like this:
  ---------- Timing report ----------
    hostgroups: min 1, max 100000
    rtt-timeouts: init 250, min 50, max 300
    scan-delay: TCP 5, UDP 1000
    parallelism: min 0, max 0
    max-retries: 2, host-timeout 900000

o Modified the WinPcap installer file to explicitly uninstall an
  existing WinPcap (if you select that you wish to replace it) rather
  than just overwriting the old version.  Thanks to Doug Hoyte for
  making this change.

I created an NSIS winpcap installer that has some advantages over the official one. Most notably, it can install itself "silently" (without any user interaction) and it doesn't prevent the user from installing older versions.

Nmap 4.03

o WinPcap 3.1 binaries are now shipped in the Nmap tarball, along with
  a customized installer written by Doug Hoyte.  That new WinPcap
  installer is now used by the Nmap self-installer (if you request
  WinPcap installation).  Some Nmap users were uncomfortable with a
  "phone home" feature of the official WinPcap installer.  It connects
  back to CACE Technologies, ostensibly to display news and (more
  recently) advertisements.  Our new installer omits that feature, but
  should be otherwise perfectly compatible with WinPcap 3.1.

NMap 4.00 was released! I made some silly mistakes regarding endianess and windows dns server locations but the helpful folks on the nmap-dev mailing list helped straighten it out.

Nmap 4.01

o Fixed a bug that would cause bogus reverse-DNS resolution on
  big-endian machines.  Thanks to Doug Hoyte, Seth Miller, Tony Doan,
  and Andrew Lutomirsky for helping to debug and patch the problem.

o Fixed --system-dns option so that --system_dns works too.  Error
  messages were changed to reflect the former (preferred) name.
  Thanks to Sean Swift (sean.swift(a)bradford.gov.uk) and Peter
  VanEeckhoutte (Peter.VanEeckhoutte(a)saraleefoodseurope.com) for
  reporting the problem.

o Applied a patch for a Windows interface reading bug in the aDNS
  subsystem from Doug Hoyte.

Nmap 3.99

o Integrated all remaining 2005 service submissions.  The DB now has
  surpassed 3,000 signatures for the first time.  There now are 3,153
  signatures for 381 service protocols.  Those protocols span the
  gamut from abc, acap, afp, and afs to zebedee, zebra, and
  zenimaging.  It even covers obscure protocols such as http, ftp,
  smtp, and ssh :).  Thanks to Version Detection Czar Doug Hoyte for
  his excellent work on this.

My Asynchronous Reverse DNS resolver made it into the official nmap distribution! Here's the CHANGELOG entry:


o Reverse DNS resolution is now done in parallel rather than one at a
  time.  All scans of large networks (particularly list, ping and
  just-a-few-ports scans) should benefit substantially from this
  change.  If you encounter any problems, please let us know.  The new
  --system_dns option was added so you can use the (slow) system
  resolver if you prefer that for some reason.  You can specify a
  comma separated list of DNS server IP addresses for Nmap to use with
  the new --dns_servers option.  Otherwise, Nmap looks in
  /etc/resolve.conf (UNIX) or the system registry (Windows) to obtain
  the nameservers already configured for your system.  This excellent
  patch was written by Doug Hoyte (doug(a)hcsw.org).

Note that the correct file is /etc/resolv.conf not the typoed /etc/resolve.conf!


o Version detection softmatches (when Nmap determines the service
  protocol such as smtp but isn't able to determine the app name such as
  Postfix) can now parse out the normal match line fields such as
  hostname, device type, and extra info.  For example, we may not know
  what vendor created an sshd, but we can still parse out the protocol
  number.  This was a patch from  Doug Hoyte (doug(a)hcsw.org).

o Fixed a problem which caused UDP version scanning to fail to print
  the matched service.  Thanks to Martin Macok
  (martin.macok(a)underground.cz) for reporting the problem and Doug
  Hoyte (doug(a)hcsw.org) for fixing it.

o Made the version detection "ports" directive (in
  nmap-service-probes) more comprehensive.  This should speed up scans a
  bit.  The patch was done by Doug Hoyte (doug(a)hcsw.org).

o Integrated all of the September version detection fingerprint
  submissions.  This was done by Version Detection Czar Doug Hoyte
  (doug(a)hcsw.org) and resulted in 86 new match lines.  Please keep
  those submissions coming!

Nmap 3.90

o Applied an enormous nmap-service-probes (version detection) update
  from SoC student Doug Hoyte (doug(a)hcsw.org).  Version 3.81 had
  1064 match lines covering 195 service protocols.  Now we have 2865
  match lines covering 359 protocols!  So the database size has nearly
  tripled!  This should make your -sV scans quicker and more
  accurate.  Thanks also go to the (literally) thousands of you who
  submitted service fingerprints.  Keep them coming!

o Added "rarity" feature to Nmap version detection.  This causes
  obscure probes to be skipped when they are unlikely to help.  Each
  probe now has a "rarity" value.  Probes that detect dozens of
  services such as GenericLines and GetRequest have rarity values of
  1, while the WWWOFFLEctrlstat and mydoom probes have a rarity of 9.
  When interrogating a port, Nmap always tries probes registered to
  that port number.  So even WWWOFFLEctrlstat will be tried against
  port 8081 and mydoom will be tried against open ports between 3127
  and 3198.  If none of the registered ports find a match, Nmap tries
  probes that have a rarity less than or equal to its current
  intensity level.  The intensity level defaults to 7 (so that most of
  the probes are done).  You can set the intensity level with the new
  --version_intensity option.  Alternatively, you can just use
  --version_light or --version_all which set the intensity to 2 (only
  try the most important probes and ones registered to the port
  number) and 9 (try all probes), respectively.  --version_light is
  much faster than default version detection, but also a bit less
  likely to find a match.  This feature was designed and implemented
  by Doug Hoyte (doug(a)hcsw.org).

o Added a "fallback" feature to the nmap-service-probes database.
  This allows a probe to "inherit" match lines from other probes.  It
  is currently only used for the HTTPOptions, RTSPRequest, and
  SSLSessionReq probes to inherit all of the match lines from
  GetRequest.  Some servers don't respond to the Nmap GetRequest (for
  example because it doesn't include a Host: line) but they do respond
  to some of those other 3 probes in ways that GetRequest match lines
  are general enough to match.  The fallback construct allows us to
  benefit from these matches without repeating hundreds of signatures
  in the file.  This is another feature designed and implemented
  by Doug Hoyte (doug(a)hcsw.org).

o Fixed a crash found during certain UDP version scans.  The crash was
  discovered and reported by Ron (iago(a)valhallalegends.com) and fixed
  by Doug Hoyte (doug(a)hcsw.com).

o Added "Exclude" directive to nmap-service-probes grammar which
  causes version detection to skip listed ports.  This is helpful for
  ports such as 9100.  Some printers simply print any data sent to
  that port, leading to pages of HTTP requests, SMB queries, X Windows
  probes, etc.  If you really want to scan all ports, specify
  --allports.  This patch came from Doug Hoyte (doug(a)hcsw.org).

nmap-3.81-p-switch-additions.patch and nmap-3.81-factor.patch
These patches add some interesting (and in my opinion useful) features to the -p switch:

Although these aren't included into the official nmap distribution, I'd like to port them to the latest version of nmap and try to get them included some day.

Nmap ARP Scanning Patch
On a very old version of Nmap, I implemented a new ping type: ARP ping. I chose -PR as the command line switch. This patch requires libnet to run and is completely useless now with the introduction of ARP scanning (-PR) in Nmap 3.90. See the included README for more info on this patch.

Nmap MP3 Player Patch
This is a joke patch never meant to be a part of the actual nmap distribution.

Nmap 2.54BETA27

o Fixed bug that caused "adding open port" messages to be printed even
  when verbose mode was not specified. (patch sent by Doug Hoyte (
  doug(a)hcsw.org ).