Author Topic: The development of nodesn - a sniffer + web gui for monitoring+analyzing attacks  (Read 8702 times)

0 Members and 1 Guest are viewing this topic.


  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email
gui has always been my down fall.. i need to write alot more test codes & focus on the actual backend side of this project.

ever since i retired nodesn and began rewriting it as dnode, ive been stuck.. and it's entirely due to the fact that im trying to focus on gui 100%.

backend shell-like client stuff:
# enter agent
agent# exit
# ...
- translate these inputs into the json/object literals

- access
- each agent gets a group name
- each agent gets a group_name for sub groups, ie, adarq.org_routing
- each policy gets a read, NONE, or RWX permission
- main groups provide global access to all services, but are superceded by more fine grained access
- always check none first, deny before allow
- you can login as an agent to set user/group permissions for that agent
- any time you modify a policy, or anything for that matter, save a copy, deactive copy, activate new copy with new modification


policies table:
  • id: id of the policy, auto incremented
  • name: name of the policy.. 'name' for global, 'name_subname' for more specific policy pertaining to a global, and so on.. 'agent_routing', 'agent_routing_policy'
  • mode: R, RWX, NONE
  • uid: this policy describes a user
  • gid: this policy describes a list of users by group
  • status: 0 inactive, 1 active
  • ts: timestamp on modification

every table has a 'who' value, which links any modification/addition back to the user who is responsible.
- every action is run through conf.deps.db.query(), this will allow us to monitor all actions in real time by the admin
- the timestamp on every modification is great because we can roll back or forward changes, globally or per agent... instantly..
- everything also needs a comment field, this can be used to also rollback/forward changes.
- do we need a created, modified, disabled timestamp ?? instead of just one 'ts' (modified) ?
- global changes should effect every agent

rules need to be as granular as possible.. if we want to grant ALL access to a specific firewall rule and that's it, then it should be possible.

all variables to functions need to be validated or var'd at the top of each function... also, a comment needs to be listed per function, to explain usage/arguments etc.

example policies & sub policies:


very small agent example:
adarq.org_routes_static_192.168.1.1 <-- ability to edit, disable, etc

                b: self.c,
                str: str,
                cb: self.insert_cap_summary_cb_save,
                xself: self,

Much of these posts will be 'modified', as they are a centralized source of info. Work in progress since i'm migrating over some info from my box.

Overall Concept

This tool is a server which sniffs the wire for attack probes & actual attacks, feeds this info to a db/worker handlers, then also feeds the info to numerous clients which are connected via web browser. The goal is to create a very clean code base which can easily be extended, and a rich set of features for monitoring traffic, analyzing traffic, investigating traffic, and allowing different levels of users to connect to the server & perform operations based on their access credentials. For example, guest access would sanitize all ip addresses and remove the ability to set blocks via iptables rules etc.

EDIT: Remember adarqui, this is a simple tool, make sure not to get caught up in b.s. functionality. Primary concerns are logging properly, searching, visual graphs of the attacks (custom and general).

  • Developed in nodeJS server side
  • Developed in standard javascript client side - REAL TIME Websockets ui
  • Very modular code framework that allows for easily incorporating new modules & concepts with minor code changes
  • Sniff traffic on specified interfaces.
  • Query a database for per-interface filter strings, exceptions, and the like.
  • Insert sniffed packets into multiple MYSQL tables: cap, cap_summary, & info
  • For uniquely new attacks, if infoGathering is enabled, perform various nmap, openvas, geoip, whois, traceroute, etc tools on the source ip
  • Possibly interface with a snort unix-socket in order to receive IDS alerts, correlate to attacker in SQL DB
  • Provide a clean interface to the user for monitoring packets in real time as well as sorting, searching, filtering, blocking via iptables, etc.
  • Provide a clean tools interface, for gathering info on attackers or other ip's
  • Support ipv6

x = Pretty much done
? = Working on it
- = havn't started working on it


  • x Modular Framework
  • x Dynamic navbar builder, using tabs or accordion
  • x HTTPS server only
  • x Authorization first : Took this out temporarily
  • - GZIP encoding
  • - Fun twitter/facebook post tool
  • - Incorporation of nodejs's brain module : This program should LEARN why a blok was set, ie, if something is always blocked because it's just really high traffic, it should pick up on this
  • - Very important, transparent database access: wrapper's for mysql, mongo, sqlite, berk db, etc
  • - Initialization mode, which initializes a db, creates tables/initial access restrictions etc
  • - incorporate custom node showbgp module
  • x Pull login div from somewhere on disk and emit it to client
  • x Auth needs to use my html5 storage + db technique
  • ? Need the nmap/traceroute/google/whois/bgp/etc data in the cap_summary table. On a new attack, launch the recon probes
  • - Need to cache/store all manual lookups into the cap_summary table
  • - How to not always send whois/geoip/nmap/bgp etc info with every summary alert? how about, when the summary packet arrives, once someone clicks any of those, it requests the entire row in the table
  • ? For dashboard: geoip update, dns update, etc... ability to update all hosts in DB that havn't been updated with the new functionalities of the code
  • - auto traceroute idea: map all AS #'s/geoip etc OF the hosts in the traceroute, for creating very interesting maps.. also, what if i fucking spidered this? NUTS..


  • - Liquid traffic shaper using CANVAS
  • x Don't reload navbar + modules on auto-reconnect (after server disconnect) - update sockets to new socket & re-use mods/dom
  • x Ability to enter one ip in a tools tab, and have EVERY tool function on that ip.. Perhaps this tool could be called "central"
  • x Need a checklist for central, so i can disable certain tools from the mass exec
  • - need nmap xml parser
  • ...


  • x GeoIP of host/ip's
  • ? Whois of host/ip's - Somewhat done, need to enable referral lookups
  • ? Google trace
  • x NMAP scan
  • x Traceroute
  • - OpenVAS Scan
  • - blacklist/offending IP searches
  • - BGP info tool : simply querying for info on what is advertised and by what version of ip etc
  • - anomaly detection without deep inspection
  • ? Pretty graph generators for everything
  • - Ability to save graphs/info as pdf/documents
  • - IP PREFIX tool etc, use the one from my other project

  • - On initial usage of nodesn, run in initializer mode
  • - Start a web browser to configure settings, admin user, which type of database to access (and credentials for that)
  • - Save this to a config file for nodesn to parse on startup

  • - Various types of graphs to model the attack data.. highcharts, sigma, etc
  • - Ability to create custom graphs based on search
  • - Ability to search for multi ports/addys etc, using an addition pane.. kinda like, add an ip to it, starts filling up, will search for all of that and graph it
  • - Most used ports, popular protocol, most attacks from, etc.
  • - DONUT CHART: protocols inside, ports outside
  • x REGULAR PIE CHART - countries
  • x REGULAR PIE CHART: protocols
  • x REGULAR PIE CHART: ports
  • - live updating maps & 'flows'.. live updating + plotting of attacks per lat/lon/country etc, plus ability to see the traceroutes of each attack (lines drawn)
  • - traceroute mapper: geoip each hop, build a set of points, connect the dots on a nice svg map
  • - Network Bandwidth, network connections etc, netflow stats?

  • x Adding geoip callback to pcap_subs.js and xgeoip.js - need all geoip fields in the table
  • - need to implement traceroute fix, whoisfix, nmapfix, etc

Interesting transition..... Possibly integrate this into the whole darqbot concept.. Re-build darqbot in node.. Incorporate tons of diff modules & come up with lots of real-time analysis/graphs etc. dno.. maybe keep this project to itself at least.. darqforum has to be nuts.


  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email
General Program Flow

Improve this! This is from my raw .txt file, need to start really focusing on these program flows & code documentation


   setup config object
   c.o.boot.load() : log boot operation, run bootup stuff

      boot_mysql()      : connect to mysql server
      process_argv_env()   : process argv & env. initializes c.o.pcap.ifaces list
      boot_express()      : config the express server
         sets up /views/public etc

      boot_https_server()   : config & boot the https server
 :  https server now running.   

            b.sock_io_app.sockets.on('connection'..) : setup new connection event handler

               new_connection object created
               new_connection.handle_connection() : handle a new connection

                  server_pulls() : pull in the server private modules, these will set their own handlers

                     loop through b.server_pulls list

                     require each of those modules
                     run the load({socket:this.socket}) operation   

                  socket.on('disconnect', and 'error') ...

   c.o.db.pcap_subs.load()   : load the pcap_subs module
   c.o.db.load()         : loads the db module
   c.o.db.boot()         : boots the sql module.

      get_filters()      : pulls the pcap filters from the SQL DB & calls the pcap_init callback

         pcap_init cb   : initializes the list of pcap interfaces to sniff on

            pcap_session.on('packet') : sets the packet hook/callback (pcap_subs.log_packet())

               pcap_subs.log_packet() : parses packet, notifies clients, adds to sql db etc



   pcap_pseudo object: Contains captured packet info

      self.constructor: called on new object creation, parses a received packet

   pcap_init: called as a callback from get_filters(), sets up event handler for pcap packets on ifaces

      pcap_session.on('packet', log_packet...) : upon receiving a packet, handle & log it

         log_packet: create a new pcap_pseudo object, parse the packet into it, log/handle it

            insert_cap_summary(): insert only unique packets into the cap_summary table
            insert_cap(): log every packet into the cap table

            notify_clients_of_packet(): for every websocket client listener, notify them of this packet

               loop through sock_io_app.sockets: emit packet to them


  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email
Bugs that arrise.

            throw new Error("Don't know how to process TCP option " + raw_pack
    at pcap_pseudo (/home/root/admin/projects/nodesn/private/js/pcap_subs.js:261:14)
    at Object.log_packet (/home/root/admin/projects/nodesn/private/js/pcap_subs.js:272:12)
    at Pcap.pcap_init (/home/root/admin/projects/nodesn/private/js/pcap_subs.js:308:21)
    at Pcap.EventEmitter.emit (events.js:96:17)
    at packet_ready (/home/root/admin/projects/nodesn/node_modules/pcap/pcap.js:63:12)
    at IOWatcher.pcap_read_callback [as callback] (/home/root/admin/projects/nodesn/node_modules/pcap/pcap.js:68:36)

An error with too many open files. This popped up when i was writing the xgeoip_fix routines. Tons of async callbacks all operating on files causes the nodejs geoip module to fail once it can't open the file due to file limits being reached... This fixes it:

Code: [Select]
ulimit -n 1000000


  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email


Eventually I need to:
- move my DEFINE_LIKE_STUFF into objects, ie:
Code: [Select]
login = {
 req: {
  LOGOUT: 2,
exports.login = login;


Also, need to get rid of all repetitive if statements/switch case statements.. move everything to a index callback:
Code: [Select]
cb = {
 blah: some_function,
 blah2: some_function2,


Also need to fix my modules bigtime... no more template code, instead, extend a base module... need COMPLETE consistency among naming conventions/functions/etc..

  FROM cap_summary
 WHERE ts >= UNIX_TIMESTAMP('2013-01-01 00:00:00')
   AND ts <  UNIX_TIMESTAMP('2013-01-11 00:00:00');

SELECT * FROM cap_summary

FROM cap_summary
WHERE ts >  '2013-01-07 00:00:00'

FROM cap_summary
WHERE ts >  '2013-01-07 00:00:00' AND ts < '2013-01-08 00:00:00'

sort by:
country code

host + cnt
country code + port
country code + cnt
asnum + port

SELECT ip_proto, count( * )
FROM cap
GROUP BY ip_proto
ORDER BY count( * )
LIMIT 0 , 30

SELECT tcp_dport, count( * )
FROM cap
GROUP BY tcp_dport
ORDER BY count( * ) DESC

SELECT udp_dport, count( * )
FROM cap
GROUP BY udp_dport
ORDER BY count( * ) DESC

SELECT count(*) from cap_summary WHERE xgeoip_asnum IS NOT NULL
SELECT count(*) from cap_summary WHERE xgeoip_asnum IS NOT NULL

describe cap_summary

highcharts svg:



  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email

This is a new feature to extend the functionality of nodesn. So, nodesn monitors a common tunnel, one interface where everything is natted through a vpn etc from external sources. So, these nodesn_agent's are going to be host-based monitoring tools that send processor usage updates, logs, etc back to the central nodesn monitoring tool. Each agent will have a login/password associated with it and an account type of "agent". Each agent that logs in will be 'enabled' in some agents tab, otherwise it'll be grey'd out as inactive.

Agent features
  • - When turned on, agent will send load avg/processor info at the specified interval (1s for example)
  • - When turned on, agent will send specific log info - every time an entry hits a log, agent will report it
  • - Upon startup, agent should send nodesn the last ~5-10 entries in each configured log
  • - Need a process accounting AND auditd parser for real-time command monitoring
  • - Process/memory usage graphs etc
  • - snort feed
  • - auditd parser
  • - ability to pick and choose which files to monitor
  • - centralized ossec log viewer
  • - CentralAgents -> ability to see all agents on same pane. ie, if u click LoadAvg in central, you get graphs of ALL of the agents on one screen, updated realtime
  • - Ability to pop a shell on any agent machine, instantly
  • - Ability of agent to run non-root

Agent features
  • - Free memory graph... need to combine multiple series on same graph? for a few at least
  • - Load avg graph
  • - Number of processes graph
  • - memory/processor usage per top 10 processes
  • - Need a 'last heartbeat' field next to each agent, current time - last heartbeat time = x sec
  • - bandwith (RX+TX) on the same graph)
  • - save all interval stuff to a table? cpu info, bandwidth info, etc? so can play it back/associate with date etc
  • - abilitty to play/pause each agent module, ie, stopping a bandwidth monitor, resuming, etc
  • - ability to set global allow/deny features for agent mods, ie, adding bandwidth monitor to disabled by default, enable per agent by request
  • - label my graph
  • - ability to customize/tweak various aspects of any agent module, ie, enabling/disabling various interfaces to be monitored (fortigate arrow over style would be kinda cool)
  • - auditd exec parser
  • - process accounting parser
  • - make this multiOS, for the agents, they should support BSD, linux, and solaris.. need hooks based on Platform..
  • - need to add lots of /proc info to agent->profile tab, such as ip_forward, etc.. stuff like that
  • - need custom interface and shell.. custom interface can be root, shell should be dropped privs
  • - each menu in an agent can have: READ, READ/WRITE, and NONE perms... for none, users cant even access that section

  • - system -> usage: per ip bandwidth usage, cli console, system information, logs and archive stats, system resources (gauges), top sessions by destination address
  • - system -> network: interaces, administrative status (up or down)
  • - system -> dhcp server -> create new
  • - system -> monitor -> ..
  • - router -> static route -> create new -> dest ip/mask, device, gateway, comments, priority, distance
  • - router -> policy route -> if incoming traffic matches: protocol, incoming interface, source address/mask, destination address/mask, destination ports, type of service ... force traffic to: outgoing interface, gateway address...
  • - router -> settings -> new gateway detection -> interface, ping server, detect protocol (icmp ping, tcp echo, udp echo), ping interval, failover threshold, HA priority 1
  • - router -> dynamic -> rip, ospf, bgp ,multicast
  • - router -> monitor -> list of all routing protocols
  • - ability to set alert thresholds etc, say for cpu processor utilization, free mem, bandwidth, etc.. can be alerted via web interface or email
  • - on each agent, have a 'users list', any time someone navs to that agent, send an op, then notify anyone else on that agent
  • - use session id's, so that, if anything is changed by someone else, it notifies other users who are working on the same agent

all interface things should be selection menus.

for snort logging:
snort -c /etc/snort/snort_nodesn.conf

in /etc/snort/snort_nodesn_agent.conf (get rid of the other output logging)
Code: [Select]
output alert_csv: /var/log/snort/alerts.csv default

example output:
Code: [Select]
01/18-00:09:39.913468 ,1,100000160,2,"COMMUNITY SIP TCP/IP message flooding directed to SIP proxy",ICMP,,,,,AA:0:4:0:A:4,0:17:F2:E9:B6:EC,0x62,,,,,,64,0,60776,84,86016,0,0,9345,283
01/18-00:09:39.918824 ,1,100000160,2,"COMMUNITY SIP TCP/IP message flooding directed to SIP proxy",ICMP,,,,,0:17:F2:E9:B6:EC,AA:0:4:0:A:4,0x62,,,,,,64,0,0,84,86016,8,0,9345,287

• timestamp • sig generator • sig id • sig rev • msg • proto • src • srcport • dst • dstport • ethsrc • ethdst • ethlen • tcpflags • tcpseq • tcpack • tcplen • tcpwindow • ttl • tos • id • dgmlen • iplen • icmptype • icmpcode • icmpid • icmpseq

databound shuttle:



nice forms!


  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email
Ok, i've hit phase 2 of this project... which is the refactor/cleanup process.. I think i'm going to completely re-do the UI using extJS (which is amazing).

the project has expanded so much that I really need some very powerful module design.. it's decent right now but, too repetitive.. plus the UI is lacking.. extJS is going to take this ui to the next level.

excited.. refactoring is always painful but every time i do it, things improve bigtime.. this first refactor i'm going to try and nail down the core ui/module structure.



For NMAP xml (or others), incorporate xml tree:

main page, initial console?

internal mail messages etc:
something like this for tools?
for saving alerts, moving interfaces over, etc?
something like this for storing alerts/pcaps:
for searching/viewing packets, operations, logs, etc:!/example/grid/infinite-scroll.html

good examples:
haha sick:
dynamic forms examples:
progress bars:
buttons examples:


extjs 4 documentation url:
DOM events:!/guide/mvc_pt1

jquery ui treeview


  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email
im having the hardest time refactoring this code... this is so brutal.


  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email
{ ctx: '', cmd: 'get', op: { chan: 'login', user: 'hi', pass: 'password' } }

all module operations are added to the various set,get,exec,show,help cmd objects..... then, based on the context, they are verified against policies and either run (server) or sent to an agent to run.

loads all modules in conf.server, conf.agent, etc..... no per sock instances of modules?? are we going to always pass a sock object to every func? since we're not creating new objects?

ability to create sql tables/rows based on the various module's inc.js's
- this can become very useful.

server should only contain an auth/login module..... the rest can be provided by running an agent ON THE SERVER SYSTEM
- duh
- SERVER = login, policy verification/relay, database, centralized server for agents to connect
- AGENT = connect to server, accepts requests from server, performs all type of services, replies back to server, server relays to clients (if needed)

everything req/resp needs an ID+SEQ, not just username..

server now becomes only worried about:
- login/auth
- restricting/permitting users/agents based on policies
- synchronizing requests/database modifications to agents and/or users

agent is now:
- connecting to server
- services entirely

simplify this FUCKING SHIT.

bleh get hardware
- from bleh = agent
get hardware
- from server

i need better 'structures', they need to be something like:

requests {
 login {
  channel: "login",
  values: {
   user: {
    type: "string",
    desc: "Username to authenticate",
   pass: {

(general) desc's will be used for keeping better notes, also, since ill have set/get/show/debug/exec commands, this stuff will come in handy for the help/format commands.. ie,

 format login request login
 command module something channel

can return the above structure...

 show login <-- shows login policy config
 exec login ... <-- exec'd a login request ie
  exec login login admin pass

each module now needs an associated:
 parse.js <-- cli string parser... take a 'string' and turn it into properly formatted structs

when you login to dnode, you are in the global view.. to enter agent views, you need to run an 'enter' command
 - server will keep track of your 'view state', ie, global or agent.. or global=server, server view, agent view
 - commands will be 'executed' based on that current view... ie,

 - # get hardware info <- get server hardware info
 - # enter
 - get hardware info <- get hardware info
 - help hardware <-- returns help info for hardware module
 - help hardware nics <-- returns help info for hardware network interfaces
 - exit
 - #

refactoring stuff:
- has to be 100% solid before moving on to the other aspects of the code
- automatic logins, policy/group controls, agent logins ,etc
- create a bunch of different types of users with different policy restrictions, make sure they cant access services they dont have in their policy
- on second login failure, disconnec
- dont socket.on any channels that arent allowed in a user's policy
- reconnect also needs to work, also...... reconnect without reloading + auto relogin.
- also... need a revision number for each module, including a generic top level revision number.. to html5 notification user of newer code availability.

various html classes need context menus:
- for example.. class="ip", will return a custom context menu with all ip tools available etc, as well as firewall block, search, etc
- other things like, class="entry", could pull up a menu with 'edit', then this could find the specific popup editing form for that selecton

need 'safe' ip's etc:
- certain ip's should result in the code refusing to add and/or delete for some reason
- reference: debacle

- get login working fully
- get barebones panel layout working (after login)
- get a real-time updating list of users/agents panel working

CLEANUP: maybe get rid of req/resp in the template? maybe just leave it as an op..
- all req's should have one or more corresponding resp's
- ie, one request, multiple types of responses

dont hardcode module list inside of server/agent, instead, have /mods/whatever/conf.js.. dynamically read/parse all of them

write little test code apps instead of focusing on the huge gui... backend + agent + cli first.. test code apps on the side

mods/login/private/help.js exec.js format.js show.js get.js set.js



  • Administrator
  • Hero Member
  • *****
  • Posts: 33005
  • who run it.
  • Respect: +8418
    • View Profile
    • Email
isvisible techniques"

too burnt out to develop the rewrite of nodesn (now dnode)... so im just adding code to nodesn... this code is a big P.O.S but it's very useful.

today i added a notes feature

over next few days i want to add:
- a simple forum
- a chat feature

forum needs to have:
- traditional forum post/threads
- blog style posts
- note style posts
- svg art posts.. - real time? ability for multiple users to collab on a "network diagram etc"

i want to add an svg paint module to it too:w