Charles Hooper

This is the discussion list for a New London Hackerspace. Get involved!

http://lists.plumata.com/mailman/listinfo/nl-space

Got this while trying to look at AT&T's cellular plans... wacky.

FBI is back in New London

Updated: Friday, 02 Oct 2009, 8:39 PM EDT
Published : Friday, 02 Oct 2009, 8:39 PM EDT

New London (WTNH) - The FBI is back in New London.

Federal agents opened their new office Friday at 2 Union Plaza in downtown.

It had been 13 years since the FBI had an office in the city.

FBI and city officials say the new digs will help beef up law enforcement in the Whaling city.

"Together we are one together, we are stronger and together we will provide the citizens of New London with the best law enforcement services available," Chief Margaret Ackley, of New London Police, said.

"I think most people feel comfortable just knowing that they're here. People I've talked to who recognize that they're here. Some of them wanted to know what's taken place to find out why they're here. I say nothing in particular has taken place it's just the idea that they recognize the need to have someone on the shoreline and here they are," Mayor Wade Hyslop, of New London, said.

Right now about five people will work out of the New London office but there is room to expand.

The FBI's Connecticut headquarters are in New Haven.

Not a bad move, really.

Sam's Schematic Collection

The CouchDB Project

Apache CouchDB is a document-oriented database that can be queried and indexed in a MapReduce fashion using JavaScript. CouchDB also offers incremental replication with bi-directional conflict detection and resolution.

CouchDB provides a RESTful JSON API than can be accessed from any environment that allows HTTP requests. There are myriad third-party client libraries that make this even easier from your programming language of choice. CouchDB%u2019s built in Web administration console speaks directly to the database using HTTP requests issued from your browser.

CouchDB is written in Erlang, a robust functional programming language ideal for building concurrent distributed systems. Erlang allows for a flexible design that is easily scalable and readily extensible.

See the introduction and the technical overview for more information.

I'm excited for MongoDB but I'd like to see a few things happen before I can seriously consider using it. One of those things is for better 64-bit support in virtualization. The other is for the codebase to mature a bit. In particular, I am excited for the auto-sharding feature to leave alpha.

In the meantime, I'm considering using CouchDB but do not have any experience with it. I'd like to hear everyone's thoughts on it, whether good or bad.

I'd like to gather some suggestions for a basic auditctl ruleset that people can use as a baseline. For example, "Watch the password, shadow, and group files; Watch executions performed from /tmp; etc."

Any suggestions?

32-bit limitations

32-bit MongoDB processes are limited to about 2.5 gb of data.  This has come as a surprise to a lot of people who are used to not having to worry about that.  The reason for this is that the MongoDB storage engine uses memory-mapped files for performance.

By not supporting more than 2gb on 32-bit, we%u2019ve been able to keep our code much simpler and cleaner.  This greatly reduces the number of bugs, and reduces the time that we need to release a 1.0 product. The world is moving toward all 64-bit very quickly.  Right now there aren%u2019t too many people for whom 64-bit is a problem, and in the long term, we think this will be a non-issue.

7 months ago

This is more recommended reading for those that are considering using MongoDB. For some, this is a deal-breaker. I don't necessarily agree with this decision, but I do hope that 64bit support by hypervisors and VPS providers is more mature by the time Mongo matures.

What about Durability?

We get lots of questions about why MongoDB doesn't have full single server durability, and there are many people that think this is a major problem.  We wanted to shed some light on why we haven't done single server durability, what our suggestions are, and our future plans.

To start, there are some very practical reasons why we think single server durability is overvalued.  First, there are many scenarios in which that server loses all its data no matter what.  If there is water damage, fire, some hardware problems, etc... no matter how durable the software is, data can be lost.  Yes - there are ways to mitigate some of these, but those add another layer of complexity, that has to be tested, proofed, and adds more variables which can fail.

In the real world, traditional durability often isn't even done correctly.  If you are using a DBMS that uses a transaction log for durability, you either have to turn off hardware buffering or have a battery backed RAID controller.  Without hardware buffering, transaction logs are very slow.  Battery backed raid controllers will work well, but you have to really have one.  With the move towards the cloud and outsourced hosting, custom hardware is not always an option.

Requirements for web applications are also changing.  99.99% uptime is no longer the goal, people want 100% uptime as much as possible.  If you have durability through a transaction log, then you have to replay it to come back up.  If you have a master and slave in the same data center and you lose power, both will have to recover which could take 5-30 minutes.[1]

Another feature of new non-relational databases is horizontal scalability.  While MongoDB's auto-sharding is still in Alpha, we still feel this is a core component. With horizontal scalability comes many servers.  If you have a 100 node cluster, worrying about every machine is a liability.  If a machine goes down in the middle of the night, you want the system to recover as fast as possible, without human intervention.  Given that, and that a high percentage of failures are hardware, the best thing is to just mark that server as inactive, and ignore it until someone can look at it easily (could be hours or days).

Given all this, we're not saying durability isn't important, we just think that single server durability isn't the best way to get true durability.  We think the right path to durability is replication (local and remote) and snapshotting.  That's why we've spent so much time making replication fast and easy and work over wide area networks in MongoDB.

We are currently planning many more enhancements to replication to make it better.

• psuedo real-time with optional blocking for writes until on multiple servers
• replica sets instead of replica pairs
• easier to create new slaves with large data sets

Now - there are definitely some cases where single server durability is the best option.  It is on our road map, its just not on the short list right now.  We know what we want to do and how we want to do it, it's just a matter of code :)

[1] Some databases such as CouchDB use an append only model that allows for instantaneous restarts. However, this type of design usually requires compaction routines to be run periodically, so can be costly in high update scenarios.

1 day ago

This is very important to consider for anyone who is thinking about (or already is) using MongoDB.