Tags
Create a replicat set:
Step 1:
In this post , we will create a replica set and add some data to it.
1. Unpack replication.js from the Download Handout zip file.
2. We will create a four member replica set. Pick a root working directory to work in. Go to that directory in a console window.
Given we will have four members in the set, and four mongod processes, create four data directories:
mkdir -p /var/lib/mongo mkdir -p /var/log/mongodb
[hduser@base etc]$ cat mongod.conf # mongod.conf #where to log logpath=/var/log/mongodb/mongod.log logappend=true # fork and run in background fork=true #port=27017 dbpath=/var/lib/mongo # location of pidfile pidfilepath=/var/run/mongodb/mongod.pid # Listen to local interface only. Comment out to listen on all interfaces. bind_ip= 192.168.56.71 # Disables write-ahead journaling # nojournal=true # Enables periodic logging of CPU utilization and I/O wait #cpu=true # Turn on/off security. Off is currently the default #noauth=true #auth=true # Verbose logging output. #verbose=true # Inspect all client data for validity on receipt (useful for # developing drivers) #objcheck=true # Enable db quota management #quota=true # Set oplogging level where n is # 0=off (default) # 1=W # 2=R # 3=both # 7=W+some reads #diaglog=0 # Ignore query hints #nohints=true # Enable the HTTP interface (Defaults to port 28017). #httpinterface=true # Turns off server-side scripting. This will result in greatly limited # functionality #noscripting=true # Turns off table scans. Any query that would do a table scan fails. #notablescan=true # Disable data file preallocation. #noprealloc=true # Specify .ns file size for new databases. # nssize= # Replication Options # in replicated mongo databases, specify the replica set name here #replSet=repset # maximum size in megabytes for replication operation log #oplogSize=1024 # path to a key file storing authentication info for connections # between replica set members #keyFile=/path/to/keyfile #add new storage engine wiredtiger storageEngine= wiredTiger #add smallfiles to reduce the size of data files smallfiles=true quiet = false
3. We will now start a single mongod as a standalone server. Given we will have four mongod processes on our four virtual machines base,dn1,dn2,dn3.
# starting as a standalone server on the first node (IP 192.168.56.72):
sudo service mongod start
Note: for all mongod startups in the homework this chapter, you can optionally use –logPath, –logappend, and –fork. Or, since this is just an exercise on a local PC, you could simply have a separate terminal window for all and forgo those settings. Run “mongod –help” for more info on those.
4. In a separate terminal window (cmd.exe on Windows), run the mongo shell with the replication.js file:
below the code of replication.js:
// replication.js homework = { } var d = db.getSisterDB("replication"); homework.init = function() { var rsConf; try { rsConf = rs.conf(); } catch (e) { rsConf = null; } if( rs.Conf != null ) { print("at this stage of the homework the mongod should NOT be using --replSet"); return; } if( d.foo.count() != 0 ) { print("expected replication.foo collection to be empty to init. can't init. :-("); return; } for( var i = 0; i 7 ) { print("something isn't right yet. did you wait for the new members to finish synchronizing?"); } return x; } homework.d = function() { var s = rs.status(); if( s.members.length != 2 ) { print("something is wrong i don't see 2 members"); return; } var x = 0; s.members.forEach( function(m){x+=m._id+m.state} ); if( x > 9 ) { print("something isn't right yet. did you wait for a new election to finish? want to see a primary and a secondary"); } return x; }
mongo --host 192.168.56.71 --shell replication.js Then run in the shell: homework.init()
See below the example:
[hduser@base Downloads]$ mongo --host 192.168.56.72 --shell replication.js MongoDB shell version: 3.0.8 connecting to: 192.168.56.72:27017/test type "help" for help Server has startup warnings: 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files. 2016-01-27T15:46:04.416-0500 I CONTROL [initandlisten] > homework.init() ok
This will load a small amount of test data into the database.
Now run:
homework.a() > homework.a()
Step 2:
Now convert the mongod instance (the one in the node 192.168.56.72 above, which uses “–dbpath /var/lib/mongo” on our mongod config file ) to a single server replica set.
To do this, you’ll need to stop the mongod (NOT the mongo shell instance) and restart it with “replSet=repset ” on our mongod config file. Give the set any name you like.
Then go to the mongo shell. Once there, run
rs.initiate() repset:PRIMARY> cf=rs.conf() { "_id" : "repset", "version" : 19, "members" : [ { "_id" : 1, "host" : "dn1.deb.com:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : 0, "votes" : 1 } ], "settings" : { "chainingAllowed" : true, "heartbeatTimeoutSecs" : 10, "getLastErrorModes" : { }, "getLastErrorDefaults" : { "w" : 1, "wtimeout" : 0 } } }
Note: if we do not specify a configuration, the mongod will pick one based on your computer’s hostname.
When our first ran homework.init(), we loaded some data into the mongod. we should see it in the replication database.We can confirm with:
use replication db.foo.find() repset:PRIMARY> use replication switched to db replication repset:PRIMARY> show collections; foo repset:PRIMARY> db.foo.find() { "_id" : ObjectId("56a98d9e711b8b89716fc459"), "x" : 0, "y" : 0.5049361388664693 } { "_id" : ObjectId("56a98d9f711b8b89716fc45a"), "x" : 1, "y" : 0.8102677734568715 } { "_id" : ObjectId("56a98d9f711b8b89716fc45b"), "x" : 2, "y" : 0.21622777171432972 } { "_id" : ObjectId("56a98d9f711b8b89716fc45c"), "x" : 3, "y" : 0.18402799568139017 } { "_id" : ObjectId("56a98d9f711b8b89716fc45d"), "x" : 4, "y" : 0.7456490981858224 } { "_id" : ObjectId("56a98d9f711b8b89716fc45e"), "x" : 5, "y" : 0.627411370864138 } { "_id" : ObjectId("56a98d9f711b8b89716fc45f"), "x" : 6, "y" : 0.8991714371368289 } { "_id" : ObjectId("56a98d9f711b8b89716fc460"), "x" : 7, "y" : 0.05076730018481612 } { "_id" : ObjectId("56a98d9f711b8b89716fc461"), "x" : 8, "y" : 0.1309488145634532 } { "_id" : ObjectId("56a98d9f711b8b89716fc462"), "x" : 9, "y" : 0.7651711588259786 } { "_id" : ObjectId("56a98d9f711b8b89716fc463"), "x" : 10, "y" : 0.9086405225098133 } { "_id" : ObjectId("56a98d9f711b8b89716fc464"), "x" : 11, "y" : 0.7508354147430509 } { "_id" : ObjectId("56a98d9f711b8b89716fc465"), "x" : 12, "y" : 0.961310219950974 } { "_id" : ObjectId("56a98d9f711b8b89716fc466"), "x" : 13, "y" : 0.943877124460414 } { "_id" : ObjectId("56a98d9f711b8b89716fc467"), "x" : 14, "y" : 0.6795263839885592 } { "_id" : ObjectId("56a98d9f711b8b89716fc468"), "x" : 15, "y" : 0.7893385454080999 } { "_id" : ObjectId("56a98d9f711b8b89716fc469"), "x" : 16, "y" : 0.36999604175798595 } { "_id" : ObjectId("56a98d9f711b8b89716fc46a"), "x" : 17, "y" : 0.6817439894657582 } { "_id" : ObjectId("56a98d9f711b8b89716fc46b"), "x" : 18, "y" : 0.8952942125033587 } { "_id" : ObjectId("56a98d9f711b8b89716fc46c"), "x" : 19, "y" : 0.10525347525253892 } Type "it" for more
Once done with that, run
repset:PRIMARY> homework.b()
In the mongo shell and enter that result below.
Step 3:
Now add three more members to the set. We used the nodes 192.168.56.71,192.168.56.73,192.168.56.74 and there directories created in homework on step 1. Run those three mongod’s on ports 27017 with differents IP address .
Remember to use the same replica set name as we used for the first member (repset) .
We will need to add these three new members to our replica set, which will initially have only one member. In the shell running on the first member,we can see your replica set status with
repset:PRIMARY> rs.status() { "set" : "repset", "date" : ISODate("2016-01-29T19:01:17.132Z"), "myState" : 1, "members" : [ { "_id" : 1, "name" : "dn1.deb.com:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 166519, "optime" : Timestamp(1453954854, 1), "optimeDate" : ISODate("2016-01-28T04:20:54Z"), "electionTime" : Timestamp(1453950672, 1), "electionDate" : ISODate("2016-01-28T03:11:12Z"), "configVersion" : 19, "self" : true } ], "ok" : 1 }
Initially it will have just that first member. Connecting to the other members will involve using:
rs.add()
For example,
rs.add( { host: “base.deb.com:27017”, arbiterOnly:false} )
rs.add( { host: “dn2.deb.com:27017”, arbiterOnly:false} )
rs.add( { host: “dn3.deb.com:27017”, arbiterOnly:false} ).
Note that ‘192.168.56.71’ almost certainly won’t work for you unless you have already set it as ‘192.168.56.71’ in the previous problem. If not, try using the name in the “members.name” field in the document you get by calling rs.status(), but remember to use the correct port!.
You’ll know it’s added when you see an:
repset:PRIMARY> rs.status() { "set" : "repset", "date" : ISODate("2016-01-30T04:04:35.975Z"), "myState" : 1, "members" : [ { "_id" : 1, "name" : "dn1.deb.com:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 199117, "optime" : Timestamp(1454100260, 1), "optimeDate" : ISODate("2016-01-29T20:44:20Z"), "electionTime" : Timestamp(1454095399, 1), "electionDate" : ISODate("2016-01-29T19:23:19Z"), "configVersion" : 100120, "self" : true }, { "_id" : 2, "name" : "base.deb.com:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26277, "optime" : Timestamp(1454100260, 1), "optimeDate" : ISODate("2016-01-29T20:44:20Z"), "lastHeartbeat" : ISODate("2016-01-30T04:04:34.950Z"), "lastHeartbeatRecv" : ISODate("2016-01-30T04:04:34.088Z"), "pingMs" : 1, "configVersion" : 100120 }, { "_id" : 3, "name" : "dn2.deb.com:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 26303, "optime" : Timestamp(1454100260, 1), "optimeDate" : ISODate("2016-01-29T20:44:20Z"), "lastHeartbeat" : ISODate("2016-01-30T04:04:34.942Z"), "lastHeartbeatRecv" : ISODate("2016-01-30T04:04:35.855Z"), "pingMs" : 1, "configVersion" : 100120 }, { "_id" : 4, "name" : "dn3.deb.com:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 16, "optime" : Timestamp(1454100260, 1), "optimeDate" : ISODate("2016-01-29T20:44:20Z"), "lastHeartbeat" : ISODate("2016-01-30T04:04:35.031Z"), "lastHeartbeatRecv" : ISODate("2016-01-30T04:04:35.203Z"), "pingMs" : 2, "configVersion" : 100120 } ], "ok" : 1 }
document.
Once a secondary has spun up, we can connect to it with a new mongo shell instance. Use
rs.slaveOk()
to let the shell know we’re OK with (potentially) stale data, and run some queries. We can also insert data on our primary and then read it out on our secondary.
Once we have three secondary servers, both of which have sync’d with the primary and are caught up, run (on your primary):
homework.c()
and enter the result below.
Step 4:
We will now remove the first member dn1.deb.com(@ port 27017) from the set.
As a first step to doing this we will shut it down. (Given the rest of the set can maintain a majority, we can still do a majority reconfiguration if it is down.)
We could simply terminate its mongod process, but if we use the replSetStepDown command, the failover may be faster. That is a good practice, though not essential. Connect to member dn1.deb.com(port 27017) in the shell and run:
rs.stepDown() repset:PRIMARY> rs.stepDown() 2016-01-29T23:14:48.557-0500 I NETWORK DBClientCursor::init call() failed 2016-01-29T23:14:48.620-0500 E QUERY Error: error doing query: failed at DBQuery._exec (src/mongo/shell/query.js:83:36) at DBQuery.hasNext (src/mongo/shell/query.js:240:10) at DBCollection.findOne (src/mongo/shell/collection.js:187:19) at DB.runCommand (src/mongo/shell/db.js:58:41) at DB.adminCommand (src/mongo/shell/db.js:66:41) at Function.rs.stepDown (src/mongo/shell/utils.js:1006:15) at (shell):1:4 at src/mongo/shell/query.js:83 2016-01-29T23:14:48.622-0500 I NETWORK trying reconnect to dn1:27017 (192.168.56.72) failed 2016-01-29T23:14:48.625-0500 I NETWORK reconnect dn1:27017 (192.168.56.72) ok
Then cleanly terminate the mongod process for member dn1.deb.com.
[hduser@dn1 Downloads]$ sudo service mongod stop Stopping mongod: [ OK ]
Next, go to the new primary of the set. we will probably need to connect with the mongo shell, which we’ll want to run with ‘–shell replication.js’ since we’ll be getting the homework solution from there. Once we are connected, run rs.status() to check that things are as we expect. Then reconfigure to remove member dn1.deb.com.
repset:PRIMARY> rs.status() { "set" : "repset", "date" : ISODate("2016-01-30T04:19:11.508Z"), "myState" : 1, "members" : [ { "_id" : 1, "name" : "dn1.deb.com:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : Timestamp(0, 0), "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2016-01-30T04:19:09.622Z"), "lastHeartbeatRecv" : ISODate("2016-01-30T04:17:42.768Z"), "pingMs" : 0, "lastHeartbeatMessage" : "Failed attempt to connect to dn1.deb.com:27017; couldn't connect to server dn1.deb.com:27017 (192.168.56.72), connection attempt failed", "configVersion" : -1 }, { "_id" : 2, "name" : "base.deb.com:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 118, "optime" : Timestamp(1454126954, 1), "optimeDate" : ISODate("2016-01-30T04:09:14Z"), "lastHeartbeat" : ISODate("2016-01-30T04:19:09.580Z"), "lastHeartbeatRecv" : ISODate("2016-01-30T04:19:10.979Z"), "pingMs" : 1, "configVersion" : 100121 }, { "_id" : 3, "name" : "dn2.deb.com:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 27180, "optime" : Timestamp(1454126954, 1), "optimeDate" : ISODate("2016-01-30T04:09:14Z"), "electionTime" : Timestamp(1454127289, 1), "electionDate" : ISODate("2016-01-30T04:14:49Z"), "configVersion" : 100121, "self" : true } ], "ok" : 1 }
Tip: we can either use rs.reconfig() with your new configuration that does not contain the first member, or rs.remove(), specifying the host:port of the server you wish to remove as a string for the input.
cfg=rs.conf() cfg=rs.conf()
When done, run:
> homework.d()
and enter the result.
Step 5:
Note our replica set now has an even number of members, and that is not a best practice. However, to keep the homework from getting too long we’ll leave it at that for now, and instead do one more exercise below involving the oplog.
To get the right answer on this problem, you must perform the homework questions in order. Otherwise, your oplog may look different than we expect.
Go to the secondary in the replica set. The shell should say SECONDARY at the prompt if you’ve done everything correctly.
Switch to the local database and then look at the oplog:
db.oplog.rs.find()
[hduser@base Downloads]$ mongo --host base --shell replication.js MongoDB shell version: 3.0.8 connecting to: base:27017/test repset:SECONDARY> use local; switched to db local repset:SECONDARY> db.oplog.rs.find() { "ts" : Timestamp(1454100260, 1), "h" : NumberLong("7410701296929487099"), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 100120 } } { "ts" : Timestamp(1454126954, 1), "h" : NumberLong("7410701296929487099"), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 100121 } } { "ts" : Timestamp(1454127704, 1), "h" : NumberLong("7410701296929487099"), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 100122 } } { "ts" : Timestamp(1454127813, 1), "h" : NumberLong("7410701296929487099"), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 100123 } } { "ts" : Timestamp(1454127915, 1), "h" : NumberLong("7410701296929487099"), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 100124 } } { "ts" : Timestamp(1454128761, 1), "h" : NumberLong("7410701296929487099"), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "Reconfig set", "version" : 100125 } }
If you get a blank result, you are not on the right database.
Note: as the local database doesn’t replicate, it will let you query it without entering “rs.slaveOk()” first.
Next look at the stats on the oplog to get a feel for its size:
db.oplog.rs.stats()
repset:SECONDARY> db.oplog.rs.stats() { "ns" : "local.oplog.rs", "count" : 6, "size" : 582, "avgObjSize" : 97, "storageSize" : 36864, "capped" : true, "max" : -1, "maxSize" : 1038090240, "wiredTiger" : { "metadata" : { "formatVersion" : 1, "oplogKeyExtractionVersion" : 1 }, "creationString" : "allocation_size=4KB,app_metadata=(formatVersion=1,oplogKeyExtractionVersion=1),block_allocation=best,block_compressor=snappy,cache_resident=0,checkpoint=(WiredTigerCheckpoint.7=(addr=\"018481e48f44da9c8581e4ecee3cf38681e4400b6ae2808080e26fc0e24fc0\",order=7,time=1454128814,size=28672,write_gen=12)),checkpoint_lsn=(8,11648),checksum=on,collator=,columns=,dictionary=0,format=btree,huffman_key=,huffman_value=,id=9,internal_item_max=0,internal_key_max=0,internal_key_truncate=,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=0,prefix_compression_min=4,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,value_format=u,version=(major=1,minor=1)", "type" : "file", "uri" : "statistics:table:collection-0-6351499121938551993", "LSM" : { "bloom filters in the LSM tree" : 0, "bloom filter false positives" : 0, "bloom filter hits" : 0, "bloom filter misses" : 0, "bloom filter pages evicted from cache" : 0, "bloom filter pages read into cache" : 0, "total size of bloom filters" : 0, "sleep for LSM checkpoint throttle" : 0, "chunks in the LSM tree" : 0, "highest merge generation in the LSM tree" : 0, "queries that could have benefited from a Bloom filter that did not exist" : 0, "sleep for LSM merge throttle" : 0 }, "block-manager" : { "file allocation unit size" : 4096, "blocks allocated" : 16, "checkpoint size" : 28672, "allocations requiring file extension" : 1, "blocks freed" : 4, "file magic number" : 120897, "file major version number" : 1, "minor version number" : 0, "file bytes available for reuse" : 16384, "file size in bytes" : 36864 }, "btree" : { "btree checkpoint generation" : 26, "column-store variable-size deleted values" : 0, "column-store fixed-size leaf pages" : 0, "column-store internal pages" : 0, "column-store variable-size leaf pages" : 0, "pages rewritten by compaction" : 0, "number of key/value pairs" : 0, "fixed-record size" : 0, "maximum tree depth" : 3, "maximum internal page key size" : 368, "maximum internal page size" : 4096, "maximum leaf page key size" : 3276, "maximum leaf page size" : 32768, "maximum leaf page value size" : 67108864, "overflow pages" : 0, "row-store internal pages" : 0, "row-store leaf pages" : 0 }, "cache" : { "bytes read into cache" : 309, "bytes written from cache" : 2326, "checkpoint blocked page eviction" : 0, "unmodified pages evicted" : 0, "page split during eviction deepened the tree" : 0, "modified pages evicted" : 0, "data source pages selected for eviction unable to be evicted" : 0, "hazard pointer blocked page eviction" : 0, "internal pages evicted" : 0, "pages split during eviction" : 0, "in-memory page splits" : 0, "overflow values cached in memory" : 0, "pages read into cache" : 2, "overflow pages read into cache" : 0, "pages written from cache" : 8 }, "compression" : { "raw compression call failed, no additional data available" : 0, "raw compression call failed, additional data available" : 0, "raw compression call succeeded" : 0, "compressed pages read" : 0, "compressed pages written" : 0, "page written failed to compress" : 0, "page written was too small to compress" : 8 }, "cursor" : { "create calls" : 3, "insert calls" : 4, "bulk-loaded cursor-insert calls" : 0, "cursor-insert key and value bytes inserted" : 424, "next calls" : 7, "prev calls" : 5, "remove calls" : 0, "cursor-remove key bytes removed" : 0, "reset calls" : 1520, "search calls" : 0, "search near calls" : 0, "update calls" : 0, "cursor-update value bytes updated" : 0 }, "reconciliation" : { "dictionary matches" : 0, "internal page multi-block writes" : 0, "leaf page multi-block writes" : 0, "maximum blocks required for a page" : 0, "internal-page overflow keys" : 0, "leaf-page overflow keys" : 0, "overflow values written" : 0, "pages deleted" : 0, "page checksum matches" : 0, "page reconciliation calls" : 8, "page reconciliation calls for eviction" : 0, "leaf page key bytes discarded using prefix compression" : 0, "internal page key bytes discarded using suffix compression" : 0 }, "session" : { "object compaction" : 0, "open cursor count" : 3 }, "transaction" : { "update conflicts" : 0 } }, "nindexes" : 0, "totalIndexSize" : 0, "indexSizes" : { }, "ok" : 1 }
What result does this expression give when evaluated?
db.oplog.rs.find( { } ).sort( { $natural : 1 } ).limit( 1 ).next( ).o.msg[0]
Note that if you inserted many documents (more than around 500,000), your oplog will roll over and eliminate the document that you need. If this happens, you’ll need to repeat the previous problems with a set of clean directories in order to find the answer to this question.
step 6:
Set up a replica set that includes an arbiter.
rs.add( {_id: 1, host: "dn1.deb.com:27017", arbiterOnly:true} );
repset:PRIMARY> rs.status() { "set" : "repset", "date" : ISODate("2016-02-09T02:43:25.449Z"), "myState" : 1, "members" : [ { "_id" : 1, "name" : "dn1.deb.com:27017", "health" : 1, "state" : 7, "stateStr" : "ARBITER", "uptime" : 223, "lastHeartbeat" : ISODate("2016-02-09T02:43:23.795Z"), "lastHeartbeatRecv" : ISODate("2016-02-09T02:43:23.716Z"), "pingMs" : 1, "syncingTo" : "base.deb.com:27017", "configVersion" : 100128 },
step 7:
We have a replica set with two members, having decided that a total of two copies of our data is sufficient.
We are concerned about robustness, however.
Which of the following are options that will allow us to ensure that failover can occur if the first server goes down?
Add an arbiter.
Add another data bearing node.