Ask HN: How do you continuously monitor web logs for hack attempts?
153 sandGorgon 13 hrs 63
news.ycombinator.com/item?id=17014818
what is the generally accepted best practice to be monitoring web logs for anomalous accesses ?
do you guys just throw cloudflare in front and forget about it ? Or do you have engineers who work like data scientists - eyeball the logs ?
I have heard suggestions of using a firewall - but I'm genuinely curious on how do security focus companies discover things like "oh, we got attacked by Bitcoin miners from North Korea". Are there sophisticated tools that do this for you.. or is there a generally accepted practice that has evolved for even regular engineers to do this ?
P.S. I'm personally more interested in an API-focused answer, but I guess the same thing applies for websites.
tptacek 3 hrs
The answer isn't quite "don't bother" but it's definitely not "install these tools and get started".
Most of the things you can easily set up to watch for security events are looking for problems you simply shouldn't have.
For instance: you can set up fail2ban, sure. But what's it doing for you? If you have password SSH authentication enabled anywhere, you're already playing to lose, and logging and reactive blocking isn't really going to help you. Don't scan your logs for this problem; scan your configurations and make sure the brute-force attack simply can't work.
The same goes for most of the stuff shrink-wrap tools look for in web logs. OSSEC isn't bad, but the things you're going to light up on with OSSEC out of the box all mean something went so wrong that you got owned up.
Same with URL regexes. You can set up log detection for people hitting your admin interfaces. But then you have to ask: why is your admin interface available on routable IPs to begin with?
What you want is an effectively endless sink for developer logs (Splunk is fine but expensive, ELK is fine but complicated to manage), and you want to add structured instrumenting logs to your applications in all the security-sensitive places, like authorization checks (which should rarely fail), and especially a very solid, very long-term audit trail of admin-like actions, so you have some prayer of tracking insider misuse. You could instrument your SQL queries and catch any query failures --- which should also be rare. That kind of stuff. I don't think there's a good shrink-wrap package that does that.
But really: focus on removing attack surface, rather than continuously monitoring it.
koolba 3 hrs
But really: focus on removing attack surface, rather than continuously monitoring it.
You buried the lede on this one.
monkpit 2 hrs
I think you meant “Splunk” :)
tptacek 2 hrs
thx
peterwwillis 2 hrs
Or, you know, install these tools and get started:
Scanners:
- Nikto (https://cirt.net/nikto2)
- A comparison of tools from 2017 (http://sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-unified-list.html)
Web Application Firewalls:
- ModSecurity (https://geekflare.com/install-modsecurity-on-nginx/)
- Enable OWASP rules in ModSecurity (https://geekflare.com/modsecurity-owasp-core-rule-set-nginx/)
- Vulture (https://www.vultureproject.org/)
- Pay $500 to have SpiderLabs configure ModSecurity for you (https://ssl.trustwave.com/web-application-firewall)
- Nginx (https://www.nginx.com/products/nginx-waf/)
- Azure (https://azure.microsoft.com/en-us/blog/azure-web-application-firewall-waf-generally-available/)
- AWS (https://aws.amazon.com/waf/)
- Symantec (https://www.symantec.com/products/web-application-firewall-reverse-proxy)
- Incapsula (https://www.incapsula.com/website-security/web-application-firewall.html)
- Smoothwall (holy cow, they're still around? https://us.smoothwall.com/utm/)
- Shadow daemon (probably defunct, https://shadowd.zecure.org/overview/introduction/)
- IronBee (same, https://github.com/ironbee/ironbee)
Hardening guides:
- Apache (https://www.tecmint.com/apache-security-tips/)
- Nginx (https://geekflare.com/nginx-webserver-security-hardening-guide/)
tptacek 2 hrs
Don't do most of this.
andyfleming 2 hrs
Care to elaborate?
tptacek 2 hrs
You almost certainly don't want to set up a WAF.
Don't use any of those scanners, all of which look almost exclusively for vulnerabilities you don't have. Do look into getting a Burp license for every developer on your team.
By all means, harden Nginx. Don't pay SpiderLabs to do it for you.
peterwwillis 2 hrs
A WAF will stop a lot of the stupid junk that a kid with a "how 2 hax0r" book can make work a disturbing amount of the time.
tptacek 2 hrs
If any of that stuff is actually working on your site, not having a WAF is the least of your problems. Don't waste time with this stuff.
nxc18 1 hr
I assume you are writing in an attempt to persuade.
Your message is being lost/ignored as a result of your delivery.
We want to understand your message, but you aren't giving enough background. I don't trust your message because there is no supporting background. If you're going to tell me to not do something, either:
A) provide a compelling alternative (not doing $it is not a compelling alternative to $it in nearly all cases) B) explain in detail why you shouldn't do $it. Provide more details if asked.
I want to believe what you are saying and be convinced by it, please help me. (Note: I'm not trying to be a dick, I've just known a lot of people who are perpetually ignored because of this communication style and take forever to learn that lesson because no one bothers to say anything)
TeMPOraL 25 mins
I think tptacek's implicit point is this: all the problems you're trying to address with the tools listed are the problems you shouldn't be having in the first place, so, to quote, "focus on removing attack surface, rather than continuously monitoring it".
tptacek 1 hr
In this instance, I'm just interested in getting the right answer written down and moving on.
pvg 1 hr
I'm not trying to be a dick
You are succeeding without trying. You're essentially complaining some free advice does not fit your exact expectations and specifications. Which is fine, lots of free advice doesn't. Trying to pass this complaint off as some kind of favour to the person giving you free advice, though, is... less than great.
irundebian 19 mins
No, tptacek just doesn't give proper answers here because he thinks he can afford it with his status here. He could have just left links to further links/literature.
Alex3917 1 hr
If any of that stuff is actually working on your site, not having a WAF is the least of your problems.
Is this advice assuming that it's devs who want the WAF, rather than ops or management?
For devs clearly it's not what they should be focusing on. But saying something isn't worth doing because it shouldn't work is a terrible security principle.
tptacek 1 hr
I wrote a longer comment here, but decided it wasn't helping.
So, I'm just going to go with: no, if you're really building an application and care about application security, no, you shouldn't waste time setting up a WAF tool designed to help 10,000 employee insurance companies make sure their Wordpress marketing site isn't exposing some vulnerability from 3 years ago.
Not wasting time with dumb stuff like this isn't "terrible security"; every dumb thing you do exacts a cost from good things you could be doing instead.
Mandatum 29 mins
Hit the nail on the head. Those suites of tools offered for existing vulnerable software is usually written for platforms and configurations that shouldn't be applicable to 99% of companies.
It's like throwing gauze over an open wound. Some companies don't care enough to put thought into healing the wound altogether, whereas some just wrap it in gauze and hope they don't bleed out. And it usually works - at least for a while, until you get big enough/become a lucrative enough target to a persistent hacker.
It isn't a case of "more security tools == better security", it's a case of "why do I need these tools in the first place". If you aren't analyzing attack surfaces and building security into your software development process, I can only hope the code you write isn't accessible externally.
- using 'you' incorrectly here, speaking to those looking to apply these tools to their setup.
subcosmos 38 mins
"Don't invest in a home security system because your windows should be completely shatterproof"
balls187 0 mins
More like, "Don't invest in home security, because your most precious valuables are stored in a safety deposit box at a bank, and everything else is covered by home insurance."
sebcat 1 hr
A WAF will also add an attack surface, just like anti-virus software before them.
Company X, a content publisher, has a wordpress site with a bunch of plugins. They hear that the security record of such a setup is less than ideal. They buy a WAF solution to protect them.
Company Y, also a content publisher in the same space, realizes that they don't need dynamically generated content. They don't need tons of JS. They start publishing static content and what little dynamic functionality they have is well-compartmentalized, with interactions to the outside world carefully audited.
Company Y could also get a WAF, but why? It increases the attack surface, Company Y probably don't have the time/expertise to audit it, it is not clear what benefits it will have.
sandGorgon 2 hrs
This is brilliant.
Thanks !
peterwwillis 2 hrs
The problem with throwing tools at the thing is you have no idea what it's doing or how it works or IF it works, and it becomes forgotten about. But it's better than nothing in the short term while you figure out how to secure your application. Slap ModSecurity in front of your app while you go over OWASP and managing cloud infrastructure securely.
hluska 6 hrs
I assume that I suck at security, everything I have ever built is badly compromised and I'm too dumb to know it. I'm so bad at security that evil criminals actually celebrate my birthday.
Since I'm this bad, I assume that any monitoring tool I could install on the boxes themselves, up to and including all my logs are compromised. I'll usually still install monitoring tools for compliance, basic diligence and because they make wicked graphs that impress my superiors.
Since I'm that bad, I'd rather not invest 40 hours a week combing through fake logs unless I'm validating that I've taken care of the basic security fundamentals for my threat level.
Rather, I like to make sure that my security fundamentals are taken care of so that I can stop the simple attacks. If my threat/compliance model dictates, from there I'll possibly step things up into a web application firewall (WAF).
But, through all of this, I know that there are people out there who are so much smarter than me that the best I can do is little more than an annoyance for them. So, again, I like to assume that I'm badly compromised. This implies that I'm very careful about what data I keep, and I take efforts to protect my data at rest.
dsl 4 hrs
I break into stuff professionally.
There is nothing I love more than the "just throw {Cloudflare,Akamai,Fastly,etc} in front of it" crowd. If you are going to deploy a WAF [1] it needs to be at the edge of your datacenter or cloud environment. No matter what fancy feature your provider wants to sell you to fix this, there is a way to bypass it.
-
Make sure you are logging centrally, and logging the right things.
-
Make sure someone is looking at the logs, it doesn't have to be a full time person, but it needs to be someones job.
-
Depending on how big you are, hire people to break in on some regular basis. It could be anything from hiring a cheap contractor to run a bunch of automated tools, to full on 10 person adversarial teams. They will find some stuff for you to fix, but more importantly you will know whether the first two points are working based on what you see.
-
https://www.owasp.org/index.php/Web_Application_Firewall
dom_hutton 1 hr
Make sure you are logging centrally, and logging the right things.
Could you please elaborate on what some of the right things generally are?
subcosmos 33 mins
A tad off-topic, but one thing I've enjoyed doing in the past is giving my attackers something to get distracted on, allowing me to observe their behavior.
https://www.youtube.com/watch?v=smvwW3vW5rM
semanticist 4 hrs
We also use Fail2Ban like a poor man's Web Application Firewall - for a Rails app you know you'll never have legit traffic to anything 'cgi' or 'php' or 'exe', so crafting a couple of regexps to match those and then ban them works well.
We use CloudFlare so we feed the IP bans into CloudFlare's firewall through their API as well as local IPTables. (We also use CloudFlare's WAF - we like redundancy in our paranoia.) If you had more demanding requirements you could look at mod_security for Apache or Nginx.
We also implement rate-limiting using Fail2Ban, which helps to trip the dumber automated scans.
Every time Fail2Ban blocks an IP address, it emails our admin team with the matching log entries and a whois on the IP address, which we then eyeball to make sure there's nothing weird going on.
TBH, easily 90% of the IP addresses we block got caught looking for 'wp-admin/admin.php', which is probably a good reason to never use WordPress.
eli 3 hrs
Are these bots causing problems like consuming excess resources? If not it doesn't seem worth the effort to ban them.
These days I only block bots when they're e.g. filling out forms or posting junk comment that a human has to moderate.
PeterisP 1 hr
Some kid running an automated vulnerability scan can be automatically detected and blocked this way, which can protect you if that scan would have succeeded otherwise. Of course, the proper way would be just to scan all that yourself and actually fix the problems, but that may be too difficult in some cases.
_Chief 4 hrs
I like this approach. Do you permaban the IPs? because a ton of ISPs rotate IPs across users
inopinatus 3 hrs
Just be aware when crafting those regexes that user-generated URLs (e.g. slug-based content paths, or even just query-string params if one is very careless) can thereby be used to build a denial of service attack against internal infrastructure, third-party integrations/webhooks, and even regular customers.
Reminds me that back in the day we used to be able to train email sanitizing appliances for false positives by sending them EICAR or GTUBE with spoofed headers.
loblollyboy 5 hrs
In short - security is such a big and multifaceted topic that it’s hard to know where to start, but I think that learning Kali Linux tools would cover a lot.
I started working at a cybersecurity company - our product is ridiculously stupid and I try and work as little as possible. In the time I’m not working I try to at least learn stuff. Since I’m at a cybersecurity place I sometimes try to learn about cybersecurity. It seems that even though for profit cs companies are making crazy dough, a lot of open source solutions out there do the same thing. There are so many attacks - they can be anywhere from the application level to the level of physical devices like routers having some kind of exploit. For any type of attack there is a good or multiple good open source tools that can be used to execute or defend (depending on what color hat you have), and a ton of them come standard installed on Kali Linux, but before you learn these you have to understand networks. As a python/Django dev I didn’t really have a deep knowledge and still don’t have a deep knowledge of this. Nor have I done more than dabble with any of these tools. But I know that there are a lot of videos on how to use these - including this 15 hour (!) one https://m.youtube.com/watch?t=1177s&v=vg9cNFPQFqM - if you were to go through that you’d probably be a security wizard
khamba 56 mins
Thanks for the youtube link. Seems interesting. I am going to watch it in full.
crucialfelix 12 mins
I use logwatch. It gives a large informative report every day. Mostly it shows me that fail2ban and my secure configuration are doing what they are supposed to do.
guessmyname 3 hrs
I'm personally more interested in an API-focused answer
OSSEC [1] is my go-tool for this type of tasks, it offers a language-agnostic interface to monitor any type of events. I used it along with my team to power the early version of the Sucuri Firewall, which later became the GoDaddy's security system. The project has matured a lot and is heavily tested using millions of HTTP requests every day. We also implemented a behavior-based algorithm to detect malicious requests, this part is not open-source but I can tell you that it was relatively easy to integrate it with the core interface.
Splunk [2] is also a good option, slightly more difficult to configure though. I have had the opportunity to meet some of the developers working in this company during a short visit to their Vancouver office and was genuinely surprised with the level of engineering that goes around this product. Definitely, worth checking it.
NewRelic [3] while the purpose has nothing to do with security, it also offers an easy-to-use interface to monitor events in the system. With some extra configuration you can make it work as a rudimentary firewall and benefit from all the great features that the product provides.
Disclaimer: I am an active contributor to the core features, but I am neutral on my recommendation.
[1] https://en.wikipedia.org/wiki/OSSEC
[2] https://en.wikipedia.org/wiki/Splunk
[3] https://en.wikipedia.org/wiki/New_Relic
StreamBright 52 mins
For me monitoring the logs is already too late to do anything about the hacking attempt. Usually web servers write logs after the request was served. One approach (I have seen commercial products do it) is to hook into the web server code and inspect every request real time before it hits the application servers and make a decision if it is a legitimate request or not.
gzur 4 hrs
You don't monitor web logs, you ship the logs to cold storage every [crontab spec units for time :)] and monitor your hosts and network for unusual activity.
If anything happens, THEN you look at the logs. The signal/noise ratio of the average web host logs makes combing through them just not worth it.
There are tons of products out there - have a look at OSSEC (https://www.ossec.net/), it's open source and free.
cft 27 mins
Another question- how do you comply with GDPR and do that? What if a black hat sends you a SAR request?
gesman 4 hrs
Using Splunk to monitor traffic either via web logs or via more detailed packet capture solutions.
Just good web logs is enough to detect all kind of malicious anomalies.
Splunk machine learning functions can cluster data and detect anomalies in anything without rules or signatures:
https://imgur.com/a/IFhUZuH
wepple 5 hrs
most successful organizations ive seen don’t think about their API/app, but rather their stuff of value: usually data. If someone lifts your entire database, can you spot that on the way out the door? How quickly would you notice 100% of your fattest internet pipes spewing data at some poor souls website? You mention bitcoin; how long until you spot all your machines at 100% CPU?
twunde 3 hrs
There's really no way to eyeball logs in any sort of complex environment. What you're looking for is a system that will automatically analyze logs and/or traffic for malicious events and alert on them. What you're looking for are Introsion Detection Systems (IDS) and Security Information and Event Management (SIEM). For IDS there are a whole bunch of open source solutions. Essentially these systems ingest logs into a centralized location and run them through a rules engine, which gets updated with new rules on a weekly or daily basis.
jlgaddis 2 hrs
What you're looking for is a system that will automatically analyze logs and/or traffic for malicious events and alert on them.
(I don't pick on you, so I hope this comment doesn't come off that way.)
What are "malicious events" and how will you know them when you see them?
I would argue that, instead, your system should automatically analyse logs for benign (a.k.a., completely normal, expected, innocent) events, toss them out, and leave humans to review anything that's left over.
With any new "system", it will take a bit to build up a complete "whitelist" so that the amount of data to review is down to a manageable level but you'll be rewarded with much more meaningful and actionable data when you get there.
SnowingXIV 4 hrs
I’ve learned that at the very least to ensure that everything is being properly backed up and that the restore works. Testing this regularly puts my mind at ease that even if my site, application, whatever gets destroyed, stolen, etc I can bring it back to life in a very short time. It’s very difficult to ensure that something is 100% secure (not even sure if possible since the major players like Twitter make mistakes). Of course, this doesn’t solve the issue of data being taken but that’s why it’s important to understand the data you actually store and if you truly need to store it or if some could be handled another way.
busterarm 3 hrs
1) fail2ban 2) log everything to an ELK stack (or your preferred alternative) and set up dashboards and reports to feed you malicious crafted URLs 3) Reject traffic in your web proxy based on a maintained list of hostile URLs. Ban the source of this traffic. Don't send back 500s, just slow down and eventually stop answering the requests. HAproxy lets you do this easily with a few simple rules and a stick table.
eli 5 hrs
I used to spend a lot of time combing my logs with custom scripts, but honestly I think the effort is better spent elsewhere. There're countless automated attacks a day on a big site but they really aren't anything you need to worry about.
realandreskytt 2 hrs
SpectX is a tool that can run fast sql-like queries on top of distributed text files. It was initially built to parse logs by an infosec team. Hope it helps!
gmuslera 5 hrs
Computers are better at the eyeballs-on-logs tasks. You teach them what is "normal" to be ignored and they may warn you about what is left, even the old logwatch is pretty capable for that.
Of course, there some basic security practices. Firewall, fail2ban or similar to stop/ban misbehaving IPs, monitoring systems metrics to detect abnormal activity/usage (not just for intrusion detection), and maybe feeding logs to i.e. ELK to notice new/unknown patterns.
jenkstom 7 hrs
fail2ban, but I'm not sure that's what you are looking for. You'd have to customize it for an API.
ChuckMcM 5 hrs
Fail2ban looking for web pages that don't exist on your web site (like ../wpadmin) works pretty well at getting rid of script kiddies.
I suppose you could add a honeypot page that took form submissions and looked for any sequence attempting a CSRF, XSS, or SQL attack and ban the source there as well.
kazinator 6 hrs
I use a home-grown solution called txrban:
http://www.kylheku.com/cgit/txrban/tree/
It consists of several processes written in the TXR language. One for monitoring Apache, one for sshd and one for Exim (mail server).
TXR's standard library has "tail streams": streams which automatically follow a rotating log file.
These txrban scripts simply open tail streams on their respective log files, use TXR pattern matching to scan for what they are a looking for, and then use functions in the tiny txrban utility library to do the banning. Banning is instigated by issuing iptables commands.
The banning isn't persisted, but when you restart the scripts, they retroactively replay everything by scanning the available logs from the beginning.
Banning events are reported via syslog. A "dry run" configuration variable allows testing for the rules without actually modifying your iptables.
I never packaged this program into a redistributable form. It contains some hard-coded log-file paths that work on my server, and no documentation.
summadat 6 hrs
2nd'ing fail2ban, it's simple and awesome.
Beyond that, tripwire or it's modern equivalent. Traffic monitoring, "deep packet" inspection on the network side. Run all outbound through a proxy.
kazinator 6 hrs
Not a lot of people seem to know what tripwire is nowadays.
There is this Linux thing called IMA which in some ways reminds me of tripwire (Integrity Measurement Architecture).
tbrock 5 hrs
Nah, I’ve waned to use fail2ban multiple times and failed.
Unless something has recently changed it is some of the most poorly documented software on the planet.
I’ve never been able to configure the thing or figure out what it’s doing or... really anything besides find the homepage.
It’s also just not worth the trouble. Globally open SSH port? That’s like 1992 level Linux box in your parent’s basement stuff.
semanticist 4 hrs
While it monitors SSH by default out of the box, people here are talking about creating custom filters which will monitor their Apache or Nginx logs. (Although, that said, we also have it monitor our non-standard SSH port with v. aggressive blocking rules, even though it's also behind a firewall. Defence in depth, and all that.)
The documentation isn't the greatest, but this page explains the concepts well enough and was easy to find using Google.
https://www.fail2ban.org/wiki/index.php/MANUAL_0_8
tbrock 3 hrs
I understand the concepts but... how do you actually use it?
hluska 3 hrs
I like Fail2Ban, but as I remember it, my learning process looked a lot like this:
1.) Read through the documentation.
2.) Pour myself a scotch.
3.) Read through more documentation.
4.) Start drinking right out of the bottle.
5.) Try something.
6.) Get an error or experience a spectacular failure.
7.) Use StackOverflow.
8.) GOTO 5.
In good news, it worked - the light went on and now I'm confident that I could adapt Fail2Ban to suit any use. But, in bad news, honestly, I feel your pain and understand where/why you are getting stuck. Fail2Ban is a very complicated, yet extremely useful piece of software.
strictnein 2 hrs
I'm genuinely curious on how do security focus companies discover things like "oh, we got attacked by Bitcoin miners from North Korea". Are there sophisticated tools that do this for you..
I work in a Fortune 100 Security Operations Center (SOC). We have a dev team that builds tools for our Incident Response team, Threat Intel team, etc, etc and an engineering team to manage those tools. And lots of other teams that specialize on lots of other things (malware reverse engineering, spam/phishing, red team, etc). We have to build a fair amount of our own stuff because we operate at levels above what a lot of commercial tools can do. We have hundreds and hundreds of custom alerts that fire based on various network traffic and sysmon activity that our threat team has written rules around.
This, of course, is out of the question for most.
So, what should you do? First, get your logs all in one place. The ELK stack is probably the easiest way to do that. ES makes things searchable and Kibana is a pretty good UI for a lot of this stuff.
Second, if you aren't monitoring your internal network, you really have no idea what is going on, so look at what you can accomplish quickly with distributions like Security Onion (which includes the ELK stack): https://securityonion.net/
Third, there's a fair amount of Open Source Intel (OSINT) on threats. You'll need to find a way to integrate that information and scan your logs for it. Malware, like Coinminers, will have certain Indicators that will tell you if they are on your network. If you're on the ELK platform, look at something like ElastAlert. You write rules in Yaml and then set up Alerts to fire into your Slack channel or Email you or whatever. There is no real shortcut here.
Commercial tools that can do a lot of this stuff includes FireEye's TAP, but they definitely aren't free: https://www.fireeye.com/solutions/threat-analytics-platform....
And, as mentioned elsewhere, absolutely remove any attack surface that you can. Scan your IP ranges and domains with tools like Shodan.io, HackerTarget.com's DNS tools (DNSDumpster.com has a nice, free DNS search), etc and make sure you have a handle on what, exactly, you're exposing.
And, finally, if you're big enough then every single day a computer will be compromised in some way. There is no company in the world who doesn't experience this so you need to have a clear set of tools and playbooks to handle that. If you can respond to most incidents in days (not weeks), you'll be better than 90%+ of the companies out there.
Edit: clarified acronym
jrochkind1 2 hrs
"SoC"?
strictnein 2 hrs
Sorry, Security Operations Center: https://en.wikipedia.org/wiki/Security_operations_center
elchief 4 hrs
OSSEC is a nice FOSS tool for this
jwatte 4 hrs
What is "an attack?" You need to define this better to get more focused answers.
Log monitoring is good for finding bad access patterns, but can't be trusted to find a problem as it happens.
Look for repeated errors. Hitting the same service over an over? Login attempt failures from the same few IPs? It's either a script, or the Bing crawler. Write application level code and network level configuration to mitigate the problem.
How about legitimate users doing social engineering attacks on each other? Sybil attacks in ratings or voting? Attempt to reuse mobile purchase tokens on different accounts? Some new SSHD credential leak zero day? Using your cat picture upload service for storing pictures of dogs? Each of these need different approaches, and a good logging system that can change with your needs is super helpful.
But you can't read the logs in real time, looking for bad patterns. That needs customized analytics.
cat199 3 hrs
previously worked at a major shared webhost a few years back (which used Apache)
we used mod_security with common rulesets + a custom list which was reviewed/updated based on what hacks were coming in (with hacks both passively monitored by customer base/compromised spam reports and via active scans for malware deposited by attackers which were then cross-correlated/traced back to application logs to find the entry point/compromise.
support staff was ~600 (including front end call center and billing people), with probably 250 of those doing L2+ support and ~50 doing the security work which fed into the ruleset, for a server farm of ~2-4000ish (and exponentially more sites/accounts since it was shared webhosting). In some ways our network could have been viewed as a giant honeypot because of all of the outdated webapps that were being run by the customers, plus their compromised PC's saving passwords, etc.