This article is just for personal note/log, and don’t know if this helpful. Recently, we finish a small project(news web site) base on NodeJS / MongoDB(Replicaset) / MySQL.
servers: use Amazon EC2 m1.medium * 2 (for we don’t use cluster, just 1 core CPU is enough; but why not use small instance, it’s network issue); Theoretically, one server can handle all of this, but for more safety reason(LoadBalance), we take two.
pageview: about 5 million pageview per day (i believe it’s can handle more request)
max concurrency user: 15,000
nodejs version: 0.10.21
- huge amount news pages(directly query mongodb, and do layout) - each page at least require 50 ~ 200 mongodb query(It's a long story, that i don't wanna mention about) - each page response time approximate 50~100ms - JSONP APIs - proxy 80 port request to another port number(maybe mobile web/different domain) which listened by NodeJS too - 302+301 redirect
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
cache as possible as you can
cache everything, and let it auto expire maybe few mins; for, NodeJS is a daemon, sometime in-memory cache is better/faster than memcached server, and most important things is it no necessary to convert Object/Array <=> string each time you get and set.
so, i cache Object/Array use node-cache, but be aware on call by reference.
keep process run forever
Before, i use forever to keep process running, it’s very useful module; but currently i use init script instead. here is my example(/etc/init/web.conf):
1 2 3 4 5 6 7 8 9 10 11
prevent use JSON.parse / JSON.stringify
In my experience, after your node run about few days(approximate handle 20M requests), if you use those two method on frequently request, 100% i can sure that your node process will reach 100% CPU usage and finally can’t handle any request/response.
It’s can be trace through heapdump.
maybe few days, will become….
weird, right? i can’t explain why? but after use heapdump i found this and fixed.
my another EC2 server(m1.medium x 2) each handle 2.5M request per day, simple respose HTTP Header, calculator pageview and update mongodb fields (count++); all server after run 4 days, it’s will no response, and no any error, can’t handle any request. all i can do is just restart this service.
after remove all JSON.parse/JSON.stringify, its alive form 4 days to 1 month before last time i restart.
too many open files
log as more details as you can
my expressjs setting.
1 2 3 4 5 6 7
and my logrotate config is.
1 2 3 4 5 6 7 8 9
according to this, don’t use it.
think before use express.static
maybe nginx or apache is a good option for static file server.
Be careful on call by reference
Object or Array are call by reference, be careful.
mongoDB no open connection issue
We have 3 mongodb build with replicaset. This problem make me confuse for a long time, sometimes the mongodb connection reset, sometimes it’s show no open connection; and we tracing the wrong way either.
But all my no open connection issue come from below (i’m 99% sure, not node mongo native module issue). In my case, those two
- prevent recursive call - prevent use process.uncaugthException
Below example, will produce “no open connection”, after few seconds:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
What if, you still have to use process.on(‘uncaughtException’) for some reason ?
try process.nextTick, to escape this exception affect to mongo module, it’s useful!
wrote test case
thinking about all situation, and wrote into test case(maybe use mocha), for production service when occur unknow bug, this is quick solution to find the issue.
- such as, testing all recently URL through sitemap.xml ? if HTTP Status 200 Okay - such as, detect mognodb status, can be find data, can be use aggregate..... - such as, is MySQL/Memcached/Solr server alive?
use different log level (info/warn/error/log) and maybe add some color(red/yellow) for more easier to debug.
console.warn and console.error is stderr, it won’t wrote message when NODE_ENV=production, maybe you should override console object for this.