Results 1 to 5 of 5

Thread: Database Archiving IP Addresses

  1. #1
    Security Specialist greEd's Avatar
    Join Date
    May 2001
    Location
    Maryland
    Posts
    807

    Database Archiving IP Addresses

    Ok guys, need some help. I have an EXTENSIVE list of IP addresses in multiple flat text databases I keep from my firewall logs.
    The problem is the way the ip addresses are layed out, it simply lists the remote ip address the src port / dst port, time of probe, etc, etc.
    I'm looking for a simple way to combine each ip address into the number of probes / common probes from the ip address.

    Example:
    IP: 192.168.0.1 <number of probes> <most common port probe>

    Keep in mind that each of these files are about 40Mbs so it must be a well written program.

    Thanks.
    "I'm doing a (free) operating system (just a hobby, won't be big and professional...) for AT clones... It's not portable and it probably [won't ever] support anything other than AT hard disks, as thats all I have :-(." --Posted on Usenet August 1991 by Linus Trovalds
    http://www.computerglitch.net
    curiosity builds security | dd if=/dev/zero of=/dev/hda bs=512 count=100
    EOF

  2. #2
    SG Enthusiast
    Join Date
    Jan 2001
    Location
    DC
    Posts
    4,717
    You can do something in awk or perl to do this.

    Might make more sense to toss this data in a database and just query for reports. Would be more flexible and faster as the list sizes grow. Count the number of hits and throw the ports into a variable or a new column.

    Is this for your site? You use php and dbase access is built in. MySQL is simple to install and pretty damn fast. Don't believe it supports stored procedures yet so a lot of the calculations will have to be done in the php scrx. You could write the reports in html and make them available to you via the page (or email)

    Skye
    anything is possible - nothing is free


    Quote Originally Posted by Blisster
    It *would* be brokeback bay if I in fact went and hung out with Skye and co (did I mention he is teh hotness?)

  3. #3
    Security Specialist greEd's Avatar
    Join Date
    May 2001
    Location
    Maryland
    Posts
    807
    The list of IP addresses is actually gathered from my home setup, not my website. My site is setup with MySQL as the backend, and I have several "notification" type scripts of all traffic passing in and out of it.
    I coded a small perl script to extract duplicate ip addresses but it was way to slow for the amount of information that needs to be archived. But you do bring up a good point with mysql and php ... maybe I will install a db on my slack server and create a small php script to archive the ip addresses in mysql ... hell at least it's a start

    thanks cyberskye
    "I'm doing a (free) operating system (just a hobby, won't be big and professional...) for AT clones... It's not portable and it probably [won't ever] support anything other than AT hard disks, as thats all I have :-(." --Posted on Usenet August 1991 by Linus Trovalds
    http://www.computerglitch.net
    curiosity builds security | dd if=/dev/zero of=/dev/hda bs=512 count=100
    EOF

  4. #4
    SG Enthusiast
    Join Date
    Jan 2001
    Location
    DC
    Posts
    4,717


    Performance rule of thumb in the apache world: If you are authenticating more than a 1000 users, use a database instead of a flat file to store credentials.

    I imagine that your list of ip's that the scrx must compare against probably warrants the dbase angle.

    Be interested in hearing the results - however you handle this. I was thinking about something similar once I tidy up the project I'm working on right now.

    Skye
    anything is possible - nothing is free


    Quote Originally Posted by Blisster
    It *would* be brokeback bay if I in fact went and hung out with Skye and co (did I mention he is teh hotness?)

  5. #5
    Security Specialist greEd's Avatar
    Join Date
    May 2001
    Location
    Maryland
    Posts
    807
    ff db has never been my style when it comes to designing web sites, especially (as you stated) for user authentication, not only is speed an issue with ff, but more important to me security.

    I will post the results as soon as I get everything sound.

    regards,
    greEd
    "I'm doing a (free) operating system (just a hobby, won't be big and professional...) for AT clones... It's not portable and it probably [won't ever] support anything other than AT hard disks, as thats all I have :-(." --Posted on Usenet August 1991 by Linus Trovalds
    http://www.computerglitch.net
    curiosity builds security | dd if=/dev/zero of=/dev/hda bs=512 count=100
    EOF

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •