Analysis of nmap program flow.

main() in main.cc

Acts mainly to prepend the arguments in the environmental variable NMAP_ARGS with the command line arguments, creating a new argv and argc that is passed to nmap_main.  The conversion is done by calling arg_parse() in utils.cc.

This technique appears to have grown organically as a way to add environmental variable arguments to the argument list of the nmap command.  The old main function then became nmap_main().  I've been told that there used to be an interactive option here that would give a nmap command line and allow multiple commands to be executed one after the other in terminal mode.

nmap_main() in nmap.cc

Called only by main(), passing in argv and argc.

A fake_argv is created and duplicated from the argv we passed in.  I am not sure why this copy of argv has been created here.

The fake_argv is parsed by parse_options() also in nmap.cc.

parse_options() in nmap.cc

This function does not return a value, instead it parses the options and places the discovered items into a global data structure that the rest of the program uses.

Optimization here is to parse the commands into a data structure and return the data structure and pass the data structure to the rest of the program.   This would allow multiple commands to be ran simultaneously by the same process using different  sets of ports.

Global variable o is defined in NmapOps.cc and externally linked in just about every file in nmap.  Global variables of this type are a little bad. Variable name should reflect their scope, with only local variables having single character names.  It was actually difficult to find where the single definition was located to see what data type the variable was.

Why isn't parse opts in NmapOps.cc?  It could be built into the class.  This would put all the option related items into a single code module. 

The command line options all look like they are listed here.

After the options are parsed the code goes into a set of operations.

Replace argv with faked argv to hide which program is running.

Make sure bounce FTP site, is reachable.

Flush output for stderr and stdout.

XML output.

FakeArgv stuff.

Generate XML.

output_ports_to_machine_parseable_output(&ports);

Handle SigPIPE

Generate error if we cannot get the requested number of ports to run a parallel scan.

Debugging.

Init port class.

Randomize port.

Shortfry???


host scanning.

tcp port scan.

Raw scan (find targets?)
 Target.size stuff.

Something to do with decoy that should be fixed.


Scan by type.

script scanning



Loop through all the targets.
          Generate output xml and plain.
Free targets.

Print results.
Close everything.
Free used memory.

This function is doing a lot and going in many directions, rewriting it to hide some of the complexity such as the ultra_scan() function would make the first main function in nmap much clearer and easier to read and understand what is going on.  Might need to integrate xml generation at a lower level as just part of what our objects do.

ultra_scan() in scan_engine.cc

Sets timeout counters.

This function is very clear and clean.   Very impressive.

 while(!USI->incompleteHostsEmpty()) {
     doAnyPings(USI);
     doAnyOutstandingRetransmits(USI); // Retransmits from probes_outstanding
     /* Retransmits from retry_stack -- goes after OutstandingRetransmits for
        memory consumption reasons */
     doAnyRetryStackRetransmits(USI);
     doAnyNewProbes(USI);
     gettimeofday(&USI->now, NULL);
     // printf("TRACE: Finished doAnyNewProbes() at %.4fs\n", o.TimeSinceStartMS(&USI->now) / 1000.0);
     printAnyStats(USI);
     waitForResponses(USI);
     gettimeofday(&USI->now, NULL);
     // printf("TRACE: Finished waitForResponses() at %.4fs\n", o.TimeSinceStartMS(&USI->now) / 1000.0);
     processData(USI);
}
 

waitForResponses() in scan_engine.cc

Going to start looking here for a couple of hours tomorrow.

Hostgroups

from unnamed source high in the hierarchy of nmap:

Nmap scans hosts in groups (we call them hostgroups) which can contain 1 or more hosts, so sometimes when you scan a network,
the first group may only contain 1 host, which is done so that you can get some results back more quickly, but then the group size may increase so that it is more efficient (can scan more hosts per time period if they are done in parallel). You can see how many hosts it is scanning in the current group by pressing <enter> while Nmap is scanning

and so an ultra_scan run does one type of scan against all target ports against each host in the current group
and it can have multiple outstanding probes for multiple ports against multiple hosts (but all within the same group) in parallel  but it keeps a "congestion window" to decide how many probes it can have outstanding at once, which is based on how fast and reliable the network has shown itself to be if Nmap is getting responses quickly to all probes, Nmap can have a lot of probes outstanding at once, but if it starts seeing packet drops, Nmap slows down and reduces the parallelism

me: Interesting, so it "keeps score" as it runs   

Yes, it uses a bunch of congestion control algorithms to decide that

Scripting execution

is the .nse scripting handled by the ultra_scan function too?   

no, NSE has its own function, let me show you ...
so, first look in nmap.cc...
and you see lines like:
if (!o.noportscan) {
// Ultra_scan sets o.scantype for us so we don't have to worry
if (o.synscan)
ultra_scan(Targets, &ports, SYN_SCAN);

if (o.ackscan)
ultra_scan(Targets, &ports, ACK_SCAN);

if (o.windowscan)
ultra_scan(Targets, &ports, WINDOW_SCAN);

yeah   

so if we didn't request skipping the port scan (-sn), then it looks at the type(s) of scans we requested and
calls ultra_scan for each one. Then later in that same nmap.cc function ...
if (o.servicescan) {
o.current_scantype = SERVICE_SCAN;

service_scan(Targets);
}
so that there is version detection
then later ...
if (o.osscan){
OSScan os_engine;
os_engine.os_scan(Targets);
}
so that's OS detection 05/07/2012 04:54:57 PM

ok, I see that   

and then:
#ifndef NOLUA
if(o.script || o.scriptversion) {
script_scan(Targets, SCRIPT_SCAN);
}
#endif
so first it checks to make sure the user has liblua linked in (because it can't do NSE without LUA) and then

so the different types of scripting, version, os detection, and the like, are handled by each one of those conditionals   

it checks if the user asked for NSE with either o.script (e.g. --script) or by using version detection, which can use NSE for some of its work (o.scriptversion) then it calls script_scan with the list of target hosts in this hostgroup (Targets) to do the NSE scanning


So I could use the built in host scanning of nmap, but with my own custom OS detector script that is called on each found host?   


yeah, you could potentially have an OS detection NSE script it wouldn't be reported in Nmap's normal OS detection place, but  the output would be shown in the script output section

*nods*
that is good to know that scripts output in a different place too