Patrolling Networks For Insecurities

Mojo Jojo writes "There's a story on developerWorks about DARPA-funded work being done at Stanford Research Institute (aka SRI International) to develop soemthing called Event Monitoring Enabling Responses to Anomalous Live Disturbances (EMERALD) -- software components that are capable of providing anomaly and misuse detection for networks. EMERALD components monitor local activity, then work in conjunction with analysis engines for visualization, response, correlation, and data logging to provide a global picture of what's occurring throughout the network. Sort of like having beat cops and police call boxes throughout your network (or something)."

An opensource IDS

[lesnessman]

would be SNORT [snort.org]

Re:I dunno about this

[Faulty Dreamer]

It's probably just a 'tail -f' run through some fancy schmancy graphical environment with all the "normal" stuff filtered out. I know that tail -f 'logfile' | grep "whatever I'm looking for" has always done the trick for me. But the boss would really like it to be colorized and fancified, with buttons to click and bar graphs.

Finger a friend for world peace

[dattaway]

Back in my day, fingering was a way to be social. Reaching out and fingering someone to see if they were online for a chat after school made homework more enjoyable and productive. When fingering someone on a VMS system, I could see if they were reading their mail, what project they were working on, and what was the plan. Finger was a valuable service. Ytalk and phone with multiple connections for party lines were the motivations behind pizza parties and great study sessions.

Today, it seems finger is the product of a smear campaign to further the evils of ICQ and that AOL chat thingie or whatever they are called these days. Today's pop chat technology is a step backwards in the dark ages.

PORTMAN

[sharkey]

With its sister company,

The Patriotic Organisation for Reliable Transmission of Mysterious Additional iNformation.

--

DIAMOND

[ch-chuck]

DIAMOND

Distributed
Incoming
Anomaly
Monitor
Observing
Network
Disturbances

Re:Such Old "News"

[Anonymous Coward]

Let me start by saying that I'm posting anonymously because I work for an IDS vendor.

Not every IDS can be bypassed by using the hex escape characters. Those that can be should be dragged out into the street and shot. They're doing the equivalent of selling a door lock and saying "No one can possibly ever pick this lock. Unless they wiggle the doorknob."

Where's the value in an IDS if bypassing it is this trivial?

Such Old "News"

[tqbf]

We really need to be able to moderate story headlines. This is far beyond old news, and even when it WAS news, it wasn't interesting news.

EMERALD is not an evil government plot, nor is it interesting new technology that will change anyone's life. It's simply another research intrusion detection system, and it's been around for years. The people working on it are smart (I met and talked to Philip Porras at a Common Intrusion Detection Format meeting), but the project itself is less far-reaching than any of the commercial systems already on the market.

EMERALD is interesting primarily as a framework for building intrusion detection systems. It's component-based and designed to allow different "event generators" to be combined for analysis. This is a goal of a large number of research projects. The reason EMERALD comes up alot is that it has a relatively well-defined and powerful rule-based analysis engine to process events.

This framework differs from commercial systems like ISS RealSecure in that the sensors, which collect information from the network (or logs, or whatever) don't do the bulk of the analysis work. Unlike RealSecure, in which the raw network sniffing code is also responsible for knowing about almost every vulnerability the system detects, EMERALD allows the sniffer system to forward low-level "events" to an analysis engine that can detect attacks.

The two basic advantages to this approach is that you can more scaleably detect simple attacks and you can detect a wider range of intrusion scenarios. The system scales better because it splits the load of event generation (sniffing) and analysis (attack detection) into two components, instead of coupling them into one component like RealSecure. The system can detect more interesting attacks because it offloads analysis into a rule-based engine (basically, an "intrusion detection programming language"), so it can flexibly do things like statefully correlate different events from different event generators.

This is all nice and good, but the fact is that EMERALD is (at least, until recently) a research project with very little real-world usage. It's a nicer architecture than RealSecure, but in terms of real-world impact, RealSecure is more important; RealSecure has a fairly mature "sniffing" engine, a large database of attack signatures, and an interface that makes it easy for network operators to violate your privacy.

Anyone (the NSA, your ISP, your mother) can buy RealSecure if they have the cash. It's been available for years and years. You can deploy a RealSecure system to do everything EMERALD can currently do. And most of the interesting new capabilities EMERALD promise IMPROVE the privacy aspects of the system. You can't get a whole lot more intrusive than the "snoop every packet" semantics RealSecure already has.

So, what's the news here?

Re:Decrypting SSL?

[fizban]

I bet they mean that they catch the data before/after the encryption/decryption process within the web server itself. So at the end of the tunnel.

But I wonder... How many intruders connect to webservers over SSL when they're trying to hack into it? I guess it could help hide their tracks because external-to-the-web-server intruder detection tools wouldn't be able to see what they're doing. Maybe?

--

Re:Non-Profit Corp

[packphour]

According to their site, they are a "nonprofit corporation." Go see for yourself, at the bottom. [sri.com]

GPG password

[bogoweenie]

About 6 months ago, I wanted to take a look their eXpert-BSM thing, so I registered and downloaded the version I wanted to look at, expecting to receive the decryption password by phone or email. A few days went by and nothing happened, so I registered again. A week or so went by, so I emailed [e-mail] with a note that said something like 'I'd really appreciate the gpg password.' I stopped short of calling them because I wasn't really *that* interested.

Are there living people still working on this project? Was anyone able to get the #@%@!$ password out of them, and if so what the #$%#@ is it?

Slow down

[deran9ed]

This has been a concept for some time and in fact I tossed around the idea of a "Distributed Protection of Services" plan a while back in February to be exact and heres why most won't work.

Lazy or busy Admins
Try getting your lazy or busy admin to try and delegate programs such as this over +1000 machines and then tell him he has to trust someone else sharing this information.

Updates Updates
Name a single source entity to poll information from, it definitely shouldn't be one source since last I recalled SecurityFocus and MS had issues as does collaboration between CERT, HERT, SANS, and every other acronymed advisory board on a one shot Advisory system. (everyone wants their last words in somehow)

Policing
Who would be in overall charge of this system, the admins of the network or should they trust some shared information with others, or should we just give all trust to big brother?

Providers, ISP's, Co-lo providers
Yes competition is in a rush to share information with each other. What makes anyone think there would be uniformity when half of all ISP's, Co-lo's, and misc. providers, can't even create simple access lists on their routers let alone join together on a massive project.

Salesman Mumbo Jumbo
What about those who would take a semi nice idea and create a 'for-profit' product only to become somewhat of a PKI'ish joke where everyone thinks they need this "NEW" thing whereas theu could do fine with some other Managed Intrusion Detection Service?

DAMN!@ I bitched about nothing sign me up for 2!

Firestone Tires Spoof [antioffline.com]

Re:Hmmm... Sounds a little like 'Carnivore'...

[KenSeymour]

So TCP/IP and the Internet were originally DARPA funded.
So by your logic, they are also in the DOD, FBI, and NSA's best interest.

The tcpdump utility uses the libpcap, which was developed at Laurence Berkeley NATIONAL LABORATORY!

I suppose you will just have to balance the risk of being cracked by non-government individuals versus the risk of using US government developed network monitoring tools to protect yourself.

Nice but cumbersome,

[Ih8sG8s]

The idea is good, but I can see where this would get very cumbersome on switched networks. By definition, switched networks hamper one's ability to monitor unicast frames. Most layer two bridges use 802.1d (Spanning tree Protocol and Algorithm), which directs unicast frames only through ports required to form a single path A B connectivity. This raises issues with monitoring unicast traffic, as frames not destined for the monitor are not forwarded to the port that the monitor resides on. There's a few things you can do to overcome this:

1. Place multiple hosts into a single collision domain for monitoring of unicast traffic. This has serious performance ramifications.

2. Use an inline monitor in each collision domain where you want to monitor unicast frames.

This can be very expensive, and would get cumbersome to maintain if you have a dozen or more servers to watch.

3. Use Tap ports (available on some switches) to direct all unicast frames to a designated switch monitoring port.

This also has issues, as the tap ports are generally a low priority process in the swithcing engine, and often a simple DOS can cause the switching engine to drop packets rather than forward them to the tap port. I have also done some testing and have found that many (most) tap port services on switches are broken or selective in what traffic they forward to the tap. I can't speak for them all, but I have tested several top vendor products. I ahve a multi-homed (8) interface box that I have designed and abandoned in developing (no time) which runs linux. It's basically an 8 interface sniffer so that I can sniff up to 8 segments at a time. Even this sort of approach is really limited. Maybe they should look at a way to piggy-back patch panels in the comms room, and run a split back to an agregator so they can sniff 'everything at once' without having to deploy many, many monitors. Hey, that's a cool idea.

Re:Hmmm... Sounds a little like 'Carnivore'...

[z00t]

OMG. Who is moderating this article???

This is silly. An IDS like this monitors the nature of the traffic into/on the network, not the content of the data.

It's the difference between the network of infrared, motion, and doorway sensors of a home alarm system and a pervasive network of microphone & cameras.

NOT Already bein' done by Captus

[z00t]

The Captus product provides a subset of the functionality of EMERALD.

From the SRI EMERALD synopsis:-
EMERALD ... utilizes lightweight sensors distributed over a network or series of networks ... The EMERALD sensors monitor activity both on host servers and network traffic streams. Activity is analyzed by two independent and highly complementary analysis engines - a rule-based expert system which flags activities that match known patterns of unwanted behavior, and a statistical analyzer which flags real-time activity that deviates from the pattern of normal activity.

Captus' IDS component only utilizes a statistical analysis, and only only gathers data from the traffic stream it sees (vs. EMERALD's distributed sensors.)

That's not to say the concepts in EMERALD aren't being used elsewhere. I know ISS' IDS solution has something similar in terms of distributed sensors and central reporting. I'm not sure what methods the use for analysis, though.

hmmm

[EnderWiggnz]

thats all great and everything but look at their next product:

"URL ExchangeTM"

apparently... they have a product based on the absolute nexissity that businesses have to SHARE BOOKMARKS...

didnt know there was a market for that...

tagline

I dunno about this

[jesseraf]

SRI's EMERALD is capable of detecting the type of website attacks that plagued Yahoo!, E*TRADE and eBay in real-time.
Is that really that hard to do? Detection is easy, dealing with it is the hard part. There isn't a whole lot that detection will help you with. Just sounds like a bunch of hype to me, to lure business executives.
Jr

Hmmm... Sounds a little like 'Carnivore'...

[Bonker]

...but with a little better spin. Surely, I'm not the first paranoid to realize that government controlled or funded monitoring utilities, be they hardware-based like Carnivore, or software based like this guy, are a little scary.

EMERALD (They must have *really* worked to put this acronym together) seems on the surface to be quite a bit less scary than Carnivore. It monitors your network and reports back to you, but the project *is* DARPA funded, and ultimately serves the DOD's (and therefore the FBI and NSA's) best interests. This is the line that has me really concerned:

Plus, with resolver, an additional EMERALD software component, alerts are consolidated across multiple network domains within a single reporting console.

Does this mean that there are ways built into the software to monitor one firewalled network from another? They had better release the source for all components for reveiw, or I ain't touchin' it with a ten-foot pole. If there are backdoors in Windows, then it's just too-too easy to put a DOD or NSA back-door into something like this.

general network monitoring

[Pimpbot5000]

Emerald fits into a subset of the network monitoring that is coming of age as we speak. The IETF [ietf.org] has already begun to try and standardize protocols for use in this area...check out the Intrusion Detection Working Group [ietf.org] for more info (the results produced by the IDWG would standardize the transfer from producers to consumers mentioned in the article).

-Greg

A Screenshot

[packphour]

Least they don't try to hide their development insufficencies.

SRI Screenshot [packphour.com]

Decrypting SSL?

[Ranalou]

I have got to be misinterpreting this passage:

EMERALD security components can also help users analyze communications traffic, collecting Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP) and Web server data directly from the Transmission Control Protocol (TCP) traffic stream. "For Web traffic where we deal with Secure Socket Layer (SSL) and cryptography, we've created an embedded component to decrypt Apache Web server traffic, and we're extending it over to Netscape's Web server," Porras said.

Are they really saying that, for the purposes of intrusion detection, they will be decrypting SSL traffic off the wire and on the fly? More to the point, they're saying that this can be (relatively) easily done?

Or, is it that they're talking about an Apache module which will examine the traffic on the other side of the tunnel? The wording is a little confusing.

some stuff available

[Alien54]

Looks like some components of the beast are available for download already (Sun/Solaris only, however)

SRI plans to gradually release selected EMERALD components to the public domain. One such component, eXpert-BSM, is currently available for download from SRI's Web site (see Resources). eXpert-BSM, a small, host-based sensor that acts as a security daemon, is "particularly good for detecting misuse on Solaris operating systems," Porras said. Since SRI is a nonprofit research institute, the components made available on its Web site are released without charge to the public domain. "If we don't make certain components available on the Internet, we will still make them available to [government organizations] and to the entire DoD research community," Porras remarked.

The download is available here [sri.com].

At Least someone with some brains and experience will be able to look at it and give it a thumbs up or thumbs down.

Re:Beta Testing a New IDS

[packphour]

I looked for that link that you said not to click, but couldn't find it. Where is it? I want to see what I'm not supposed to.

Re:Beta Testing a New IDS

[Dennis Hopper]

Maybe you should browse at +1 instead of -1. That would solve the problem.

Even though you were trolling here.

Re:Such Old "News"

[Calle Ballz]

I do a lot of work with Real Secure, so I'd like to say that I know quite a bit about it's features and what it is capable of.

Any IDS that you come up with, no matter what it's features are, what it can look for, how many packets it can sniff in an hour, it is still just an IDS.

It can be bypassed easily. for ex. using %74%65%73%74-%63%67%69 in a URL rather than test-cgi.

That majority of IDS's that I have seen (including Real Secure, not sure about EMERALD though) will not detect that whatsoever.

IDS's only represent a small percentage of security and should never be relied on. The majority of security lies in Network Structure, Firewalling, keeping up to date with patches for vulnerabilities, and the KISS rule (Keep It Simple, Stupid).

Networks have insecurities just like all of us

[brianvan]

I think this is a good thing. If networks start to feel bad about themselves, we do need to figure that out and give them some counseling. They need to have self-esteem and a sense of self-worth. Otherwise, they become more prone to symptoms of depression and self-destructive behavior - such as not communicating well with their peers, carelessly dispatching backhoes to construction projects near backbone lines, etc. We need to teach them to get over their fears and self doubt, and support them so that they feel more comfortable in the presence of others. If we would just pay more attention to the warning signs of such bad feelings, we might be able to help networks before it's too late.

Have you hugged your network today?

SRI is not Stanford Research Institute

[Anonymous Coward]

They split from the university years ago. SRI is now a meaningless acronym. The "International" was tagged on to prevent a name clash, IIRC.

Re:Already bein' done

[N3MCB]

Well I worked on a project that eventualy became esecurityinc [esecurityinc.com]. I know they have been doing this stuff for a while and getting very good results. When the stuff was first being developed it drove me a bit nuts at times since I was the sysadmin on the development box and the testing set off my homegrown security scripts.

Re:Beta Testing a New IDS

[Schnedt Microne]

Indeed. I've wondered for quite awhile if at least a certain percentage of the most obnoxious trolls on this site aren't people trying to FORCE us all to read at +1 default.

I know there are those out there who resent the presence of the A.C. posting privledge, and that some of them will do whatever they can to terrorize the people who read this site into not looking at A.C. posts.

It's a 'tragedy of the commons' situation, and just a smaller version of the breakdown of the consensus model on which the 'Net as a whole is based.

SRI's patented EMERALD technology

[Don Giovanni]

Technology Overview: SRI's patented EMERALD technology complements a mix of security technologies, including firewalls, cryptography and authentication systems, virus and vulnerability scanners, providing the most comprehensive platform for intrusion detection and other network activity monitoring available today.

Re:NATALIE

[alienmole]

NATALIE=National Association of Trolls And Lamers, Idiots Every one.

Clearly

[WinDoze]

This is step one towards SKYnet. Eventually it will become self-aware and kill us all. Or even worse, become self-aware filtering software.

If we can keep it from becoming

[prisoner]

skynet and killing us all, it will be pretty cool. It does, judging from the buzz-word count in that story, sound as though it's still in the early stages:

EMERALD's intrusion detection architecture is based on software components that address real-time detection, analysis, and response for a broad range of external and internal threats. What's more, EMERALD components were designed to be independent, dynamically deployable, easily configurable, reusable, and broadly interoperable, Porras said

What year is it?

[Dancin_Santa]

I'm sorry, Dave. I'm afraid I can't do that.

Dancin Santa

NATALIE

[nspeare]

Announcing the formation of NATALIE:

The National Association for the
Termination of Acronym Labels by Idiotic Entities.

Anybody else sick of all the acronyms?

--
caution! subliminal trolling!

really...

[fjordboy]

So...chances are, all of us nerds (or geeks) will be getting tons of email for fingering, pinging and doing other normal things to websites....I think the only good way to find tampering is to go into the server logs and scroll through all of them. (yes...all 3 bajillion of them.)

Back when we didn't have network security problems

[Chuck Flynn]

I remember back when we didn't have network security problems, because we didn't have networks yet. Just standalone mainframes chugging away (back when the market was still "6 computers in the world is all we'll ever need" or however it went; my memory's a bit fuzzy these days). It goes to show you how fast the pace of technological development has come. I tell you, each new innovation in technology brings new rewards and new concerns, and network security is one of those concerns.

What was I saying? Oh yeah, I'll just reread what I wrote (how much more convenient than talking, where you don't get that option!). I remember now. Network security is a tough nut to crack, because you have to plan ahead and anticipate new attacks. No matter how much you think you're on the ball, some wiseguy will come along and show you what-for. It's like politics that way, except the stakes are more personal, since it's your own coin collection they're trying to steal. Or was it your wheel password? I can't remember.

No, wait, it's your wheel password. You kids call it "root", these days, but I like to think that not everything old should be thrown away. Where would that leave me? It's so lonely.

Already bein' done

[Charlie Bill]

I interviewed with a company a few months back that is actually doing that now:
Captus Networks [captusnetworks.com]. Damn, wish I'd actually accepted now...

Re:Acronyms

[cxreg]

I really, Really, REALLY hate it when people/organizations fit words to make a nice acronym

Then you must love the new RAM I invented! Primary Access Redundant Array Linear Logic Extended Life Overclocking Gigahertz Random Access Memory

looks familiar

[dmuntz]

Seems this will be doing something like PortSentury (sp) does. Automatically detect port scans and firewall the host. Of course, this will be on a much larger scale. Will be interesting to see if this software really works well and how it was written.

Re:Beta Testing a New IDS

[lethalp1mpslapper]

For the love of GOD!!! You are a bastard!
*ATTENTION* Please, if you value you eyes, don't click on the link above. Especially if you run Opera that saves the last open windows when closed.
You sir are EVIL!

All the "network analysis" that I need......

[Calle Ballz]

www.snort.org [snort.org]

Sniffers Rule

just an IDS

[matman]

How is this different from any good intrusion detection system? There are already companies making software like this (although I'm not aware of any open source ones)... ISS Realsecure, Axent NetProwler, NFR, Intellitactics NSM to name a few.

Maybe the big thing is that they're trying to replace the intrusion detection analyst with software... which might not be such a great idea since all (unless broken :) intrusion detection systems can generate false positives and often do.

Good Cop - Bad Cop

[spribyl]

Sort of like having beat cops and police call boxes throughout your network (or something).

Just to be a pain, this seems to be a good idea.

How do you tell a benvolent port scan from some script kiddy.
Beat cops usually wear a uniform and say hi before they strip search your car.

Re:Beta Testing a New IDS

[British]

How about self-aware slashcode that prevents anonymous cowards from posting URLS to goatse.cx and comp-u-geek along with its 2000 variants to trick you?(redirects, the IP address, etc)

Here is an amusing thought..

[redir]

One of the largest ISP's in the midwest developed a software called EMERALD about 7 years ago (still being sold, and upgraded). It was initially designed by Two guys, one was named Mike Henry from MegsInet in Chicago, IL. Needless to say they were bought out, but the software is still being sold, developed, etc...
I wonder if they are going to cause a major fuss over the fact that this software infringes on their trademark and copyright of the name, and their software also does alot of network monitoring stuff, while it certainly isn't as advanced... this new EMERALD might have to change their name to DINTCTNDBIMABB (Damn I need to check the name database before I make a big booboo)
(grin)

Re:Decrypting SSL?

[MHQ13]

It does not decrypt SSL. That EMERALD component is an apache module which can examine the HTTPS transaction *AFTER* apache has decrypted it.