鄭傑森的 "扣"

會的不多, 還在學習, 愛鬼扯蛋, 未滿40但只剩下一張嘴!

Node.js Ready for Production, Some Tips.

| Comments

This article is just for personal note/log, and don’t know if this helpful. Recently, we finish a small project(news web site) base on NodeJS / MongoDB(Replicaset) / MySQL.

status

servers: use Amazon EC2 m1.medium * 2 (for we don’t use cluster, just 1 core CPU is enough; but why not use small instance, it’s network issue); Theoretically, one server can handle all of this, but for more safety reason(LoadBalance), we take two.

pageview: about 5 million pageview per day (i believe it’s can handle more request)

max concurrency user: 15,000

nodejs version: 0.10.21

jobs:

- huge amount news pages(directly query mongodb, and do layout)
- each page at least require 50 ~ 200 mongodb query(It's a long story, that i don't wanna mention about)
- each page response time approximate 50~100ms
- JSONP APIs
- proxy 80 port request to another port number(maybe mobile web/different domain) which listened by NodeJS too
- 302+301 redirect

package.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
"dependencies": {    
  "express": "3.4.4",
  "http-proxy": "0.10.2", 
  "mongodb": "1.3.19",    
  "sequelize": "1.6.0",   
  "dateformat": "~1.0.4", 
  "flow": "0.2.3",   
  "memcached": "0.1.5",   
  "node-cache": "0.0.4",
  "consolidate": "0.8.0", 
  "swig": "~0.13.5", 
  "sprintf": "0.1.1",
  "request": "2.27.0",    
  "solr": "0.2.2",   
  "MD5": "1.0.3",    
  "xml2js": "0.2.7", 
  "log-prefix": "0.0.0",  
  "sitemap": "0.6.0",
  "mysql": "2.0.0-alpha9",
  "colors": "~0.6.2"
}

cache as possible as you can

cache everything, and let it auto expire maybe few mins; for, NodeJS is a daemon, sometime in-memory cache is better/faster than memcached server, and most important things is it no necessary to convert Object/Array <=> string each time you get and set.

so, i cache Object/Array use node-cache, but be aware on call by reference.

keep process run forever

Before, i use forever to keep process running, it’s very useful module; but currently i use init script instead. here is my example(/etc/init/web.conf):

1
2
3
4
5
6
7
8
9
10
11
start on (local-filesystems and net-device-up IFACE=eth0)       
stop on shutdown         

respawn

script      
  sleep 5          
  export NODE_ENV=production    
  cd /var/www/web
  exec /usr/bin/node --nouse-idle-notification www.js 2>&1 >> /var/log/node/www.log 
end script  

prevent use JSON.parse / JSON.stringify

In my experience, after your node run about few days(approximate handle 20M requests), if you use those two method on frequently request, 100% i can sure that your node process will reach 100% CPU usage and finally can’t handle any request/response.

It’s can be trace through heapdump.

1
var b = JSON.stringify({c:1293, t:'ads'});

maybe few days, will become….

1
b = '\\\\\\\\\\\\\\\\\...........';  // endless slash

weird, right? i can’t explain why? but after use heapdump i found this and fixed.

my another EC2 server(m1.medium x 2) each handle 2.5M request per day, simple respose HTTP Header, calculator pageview and update mongodb fields (count++); all server after run 4 days, it’s will no response, and no any error, can’t handle any request. all i can do is just restart this service.

after remove all JSON.parse/JSON.stringify, its alive form 4 days to 1 month before last time i restart.

too many open files

configure like this

log as more details as you can

my expressjs setting.

1
2
3
4
5
6
7
app.configure('development', function(){   
  app.use(express.logger('dev'));
});

app.configure('production', function(){    
  app.use(express.logger('default')); 
});

and my logrotate config is.

1
2
3
4
5
6
7
8
9
/var/log/node/*.log{  
  rotate 30 
  daily
  missingok 
  notifempty
  delaycompress  
  compress  
  copytruncate   
} 

process.on(‘uncaughtException’)

according to this, don’t use it.

think before use express.static

our new web site when lunch, first issue popup is static file server (NodeJS+express.static) not powerful, can’t handle large request on photos/css/images/javascript, even all static file send {maxAge:86000}, it’s just slowly and finally no any response.

maybe nginx or apache is a good option for static file server.

Be careful on call by reference

Object or Array are call by reference, be careful.

mongoDB no open connection issue

We have 3 mongodb build with replicaset. This problem make me confuse for a long time, sometimes the mongodb connection reset, sometimes it’s show no open connection; and we tracing the wrong way either.

But all my no open connection issue come from below (i’m 99% sure, not node mongo native module issue). In my case, those two

- prevent recursive call
- prevent use process.uncaugthException

Below example, will produce “no open connection”, after few seconds:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
process.on('uncaughtException', function(err) {
  console.log('Caught exception: ' + err);  // watch this full code trace, you will figure out why connection drop!
});

flow.exec(
  function(){
    db.find({lastName: 'jason'}).toArray(this);
  },
  function(err, docs){
    var data = [];
    for(var idx in docs){
      docs[idx].ooo = docs[idx].notExists.ooo; // will die here, then caugth by uncaughtException
      data.push(docs[idx]);
    }
    this(data);
  },
  function(data){
    res.render('index.html', data);
  }
);

What if, you still have to use process.on(‘uncaughtException’) for some reason ?

try process.nextTick, to escape this exception affect to mongo module, it’s useful!

wrote test case

thinking about all situation, and wrote into test case(maybe use mocha), for production service when occur unknow bug, this is quick solution to find the issue.

- such as, testing all recently URL through sitemap.xml ? if HTTP Status 200 Okay
- such as, detect mognodb status, can be find data, can be use aggregate.....
- such as, is MySQL/Memcached/Solr server alive?

NodeJS console

use different log level (info/warn/error/log) and maybe add some color(red/yellow) for more easier to debug.

console.warn and console.error is stderr, it won’t wrote message when NODE_ENV=production, maybe you should override console object for this.

幫你的 Cacti 加個監控告警機制

| Comments

玩了幾個月的 Node, 而且昨晚睡不著, 裝了 cacti 起來玩玩 ( 是的, 我很宅 ), 今天撞到頭的想把自己寫的爛東西 cacti-host-updown-monitor, 分享一下!

畢竟用的 node_modules 好一陣子了, 從來也沒用Github 來 OpenSource (羞~~這樣也叫 OpenSource… ) . 趁這個機會, 順便來玩玩 npm 發佈.

1
2
$ npm adduser (建立你的帳號)
$ npm publish (發佈 Node Module 到 NPM Registry)

或許你會說, cacti 不是已經有很多 plugin 了; 比如說, thold 之類的 Thresholds / Alarms 機制了, 是的, 沒錯; But i don’t care , 這部分我不想裝 plugin, 我只想用自己寫的…哈!

題外話: 現在回想起來, 比起 cacti 以前在 TTN 弄的那套 Device Monitor 還真好用(誤)….但這都是過去了…過去了…

適用對象

- 白老鼠
- 不怕死的勇者
- 有使用 cacti 做 device monitor 的人.

適用的cacti版本

- ALL

唯一的用途

- 當監控的Device Down, 就會發出告警信件通知
安裝
1
$ npm install cacti-host-updown-monitor

主程式沒有幾行扣, 呵呵…見笑了!

NodeJS 好物 Node-http-proxy

| Comments

今天要介紹一個超棒的 node_module, 小弟用了一陣子了, 覺得好用, 推薦給大家; 如果你的角色是 Server Administrator 的話, 請繼續往下看, 希望這篇簡短地介紹, 會增加你工作上的效率 (我到底在扯什麼…).

重點是, 把 port 80, 交給 NodeJS 吧

公司 R&D 狀況:

- Team A 寫 Java servlet, 提供一些啥鬼的WebService
- Team B 負責用 apache + PHP 寫 EndUser 網頁
- Team C 負責用 NodeJS 寫開放的API
- Team D 要用 nginx + php5-fpm 來寫後台控管網頁
- Team D 要用 Perl + Mason 寫統計網頁

Server:

- Server01: 8.8.8.8 / 192.168.1.1
- Server02: 192.168.1.2

Assume:

- 所有的服務都要有專屬網址
- 所有的服務只能用80 port
- domain name: jasoncheng.tw
在 server01 只要寫幾行code, 一切就幫你搞定
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
var options = {
  hostnameOnly: false,
  router: {
    // Team A
    'jasoncheng.com/servlet':   'server01:8081',

    // Team B
    'jasoncheng.tw':            'server01:8082',

    // Team C
    'api.jasoncheng.tw':        'server02:8083',

    // Team C
    'adm.jasoncheng.tw':        'server02:8084',

    // Team D 
    'adm.jasoncheng.tw/static': 'server02:8085'
  }
};

require("http-proxy").createServer(options).listen(80);

當然, 或許你會說 Nginx / Apache 本身也可以做到這些需求, 但不可否認, 人要趕流行不是嗎? 那就用 NodeJS 囉! 這麼多人都說 The Next Big Thing is Node 了.

今天, 不管開發人員要用哪些技術來寫網頁/API, 要用湯姆貓/阿帕契/引擎X當Server……都沒差; 難搞的系統管理者, 就可以趁機會跟同事們好好交朋友, 快速地幫程式開發人員, 弄一組專屬的 Demo 網址吧.

Do It Now!!
1
$npm install http-proxy

善用 HttpResponseCache

| Comments

之前寫 Android App 都要自己實作 Cache, 不管是圖片或者是API資料;

比如說: 打開程式後, 如果Local有Cache就先讀取Cache; 然後在暗地裡發送Request去更新圖片/API資料等… 當有新的, 直接複寫本地端的緩存, 然後 notify refresh. 而且還要防止Cache太多, 要定時刪除舊的資料….

太累了

使用的好處

- 節省用戶端的電力 (因為可以少掉很多Internet Connection)
- 省下龐大的頻寬費用 (因為對Server來說, 當收到 If-Modified-Since, 如果沒更新, Server端只要回應 304即可)
- 開發者不用自己再做 Cache 機制了.
- 最好的事!! 如果你本身不是用 HttpClient, HttpDefaultClient..., 而是用 HttpURLConnection的話, 你根本不用改本來的 Code.

接下來, 實作吧!! 其實很簡單, 你不必改寫你的任何Code, 你只要 Application層, 把它啓用就好了; 剩下的一切 HttpURLConnection 會幫你處理

(ResponseApplication.java) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
package com.example.testresponsecache;

import java.io.File;

import android.app.Application;
import android.util.Log;

public class ResponseApplication extends Application {
  public void onCreate() {
    super.onCreate();
    new Thread(){
      @Override
      public void run() {
        enableHttpResponseCache();
      }
    }.start();
  }
  private void enableHttpResponseCache(){
    try {
      long httpCacheSize = 10 * 1024 * 1024;
      File httpCacheDir = new File(getCacheDir(), "http");
      Class.forName("android.net.http.HttpResponseCache")
        .getMethod("install", File.class, long.class)
        .invoke(null, httpCacheDir, httpCacheSize);
    } catch (Exception e) {
      Log.e("===>", e.getMessage(), e);
    }
  }
}
(MainActivity.java) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
package com.example.testresponsecache;

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

import android.os.AsyncTask;
import android.os.Bundle;
import android.annotation.SuppressLint;
import android.app.Activity;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.util.Log;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
import android.widget.ImageView;

public class MainActivity extends Activity {

  private final String TAG = getClass().getSimpleName();
  ImageView img;
  Button msg;

  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    img = (ImageView) this.findViewById(R.id.img);
    msg = (Button) this.findViewById(R.id.msg);
    msg.setOnClickListener(new OnClickListener() {
      @Override
      public void onClick(View v) {
        new InternetTask().execute();
      }
    });
  }

  @SuppressLint("NewApi")
  class InternetTask extends AsyncTask<String, String, Boolean> {
    Bitmap bitmap;
    String jsonStr;

    @SuppressLint("NewApi")
    @Override
    protected void onPostExecute(Boolean result) {
      super.onPostExecute(result);
      img.setImageBitmap(bitmap);
      msg.setText(jsonStr);
    }

    @Override
    protected Boolean doInBackground(String... params) {
      // Test download image
      try {
        URL url = new URL("http://jasoncheng.tw/1.png");
        HttpURLConnection conn = (HttpURLConnection) (url.openConnection());
        conn.connect();
        InputStream is = conn.getInputStream();
        BitmapFactory.Options ops = new BitmapFactory.Options();
        bitmap = BitmapFactory.decodeStream(is, null, ops);
        is.close();
        conn.disconnect();
      } catch (Exception e) {
        Log.e(TAG, e.getMessage(), e);
      }

      // Test download JSON data
      try {
        URL url = new URL("http://jasoncheng.tw/1.json");
        HttpURLConnection conn = (HttpURLConnection) (url.openConnection());
        conn.connect();
        BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
        jsonStr = reader.readLine();
        InputStream is = conn.getInputStream();
        is.close();
        conn.disconnect();
      } catch (Exception e) {
        Log.e(TAG, e.getMessage(), e);
      }
      return true;
    }

  }
}
接下來用 NodeJS 寫個簡單的Static File Server
1
2
3
4
5
var express = require("express");
app = express.createServer();
app.use(express.logger());
app.use(express.static(__dirname + '/public'));
app.listen(4000);
每個Request都會產生兩隻檔案, 一個是實體檔案, 一個是 HTTP Header 資料
1
2
3
4
5
6
7
# cd /data/data/com.example.testresponsecache/cache/http
# ls -l
-rw------- u0_a48   u0_a48        345 2012-07-17 23:07 20ebbdb944f2be7d9ea96466bafe98a5.0
-rw------- u0_a48   u0_a48         42 2012-07-17 23:07 20ebbdb944f2be7d9ea96466bafe98a5.1
-rw------- u0_a48   u0_a48        321 2012-07-17 23:07 7abaca174bffb497cea054db94961804.0
-rw------- u0_a48   u0_a48      11856 2012-07-17 23:07 7abaca174bffb497cea054db94961804.1
-rw------- u0_a48   u0_a48        163 2012-07-17 23:07 journal
整個的運作的關鍵就在 Last-Modified
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# cat 20ebbdb944f2be7d9ea96466bafe98a5.0
http://jasoncheng.tw/1.json
GET
0
HTTP/1.1 200 OK
9
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/plain; charset=utf-8
Date: Tue, 17 Jul 2012 15:01:31 GMT
Last-Modified: Tue, 17 Jul 2012 13:24:14 GMT
Server: nginx/0.7.65
Transfer-Encoding: chunked
X-Android-Received-Millis: 1342537658233
X-Android-Sent-Millis: 1342537658210
第1次執行
1
2
- [Tue, 17 Jul 2012 15:14:18 GMT] "GET /1.png HTTP/1.1" 200 11856 "-" "Dalvik/1.6.0 (Linux; U; Android 4.1; sdk Build/JRN83C)"
- [Tue, 17 Jul 2012 15:14:20 GMT] "GET /1.json HTTP/1.1" 200 22 "-" "Dalvik/1.6.0 (Linux; U; Android 4.1; sdk Build/JRN83C)"
第2次執行, Server 好輕鬆, 只要回304就好了, 省下多少頻寬阿
1
2
- [Tue, 17 Jul 2012 15:14:36 GMT] "GET /1.png HTTP/1.1" 304 - "-" "Dalvik/1.6.0 (Linux; U; Android 4.1; sdk Build/JRN83C)"
- [Tue, 17 Jul 2012 15:14:36 GMT] "GET /1.json HTTP/1.1" 304 - "-" "Dalvik/1.6.0 (Linux; U; Android 4.1; sdk Build/JRN83C)"
修改1.json檔案後, Last-modified 改變了, 所以重新抓一次, 所以 Status Code 變回 200 Okay!
1
2
3
root@ubuntu:/var/www/html/jasoncheng/static_test# node web.js | grep -v favicon
- [Tue, 17 Jul 2012 15:17:30 GMT] "GET /1.png HTTP/1.1" 304 - "-" "Dalvik/1.6.0 (Linux; U; Android 4.1; sdk Build/JRN83C)"
- [Tue, 17 Jul 2012 15:17:30 GMT] "GET /1.json HTTP/1.1" 200 17 "-" "Dalvik/1.6.0 (Linux; U; Android 4.1; sdk Build/JRN83C)"

測試結果

- Okay: 關閉網路後, 圖片/JSON 資料會自己返回 Local Cache 的資料 (所以用戶不會感覺網路斷線了..嘿)
- Okay: 資料更新後, JSON會自動更新
- Okay: 檔案沒改變的條件下, Server 只回應304

測試後的問題

- Client 端必須是 Android 4.0 以後版本才有支援 (真是XD...我的NexusOne 沒辦法升到4.0啦~~~)
- Server 上的圖片修改後, Client端並不會即時更新; 懶得翻 Android Source Code了..有興趣的朋友, 再去追 (不過也沒差吧, 應該沒有人大頭照常換的吧)
- 如果遇到特殊的 port number (非80 port), cache不會生效.

2012/07/24 更新

Force a network response
1
  connection.addRequestProperty("Cache-Control", "max-age=0");
Force a cache response
1
  connection.addRequestProperty("Cache-Control", "only-if-cached");

Octopress + Github 學習經驗

| Comments

最近剛在學習使用 Octopress + Github, 把弄過幾次的經驗記錄下來.

因為我會在兩台以上的電腦使用 octopress 寫文章, 但是看過一些文章後,

覺得要在第N台電腦搞 $git clone ..這些設定, 太麻煩了.

使用 Dropbox 存放 Octopress 的資料(source..等)

當設定好 octopress $rake install 之後, 就不要再操作 git 了; ex.$git add . or $git add source,

不需要把 source 丟到 github 上面一樣可以運作

Github上面只放_depoly的資料

每次要發佈只要
1
2
$rake generate
$rake deploy

GSON - Google的兒子

| Comments

在將近3年沒寫Android的狀況下, 最近卻因為工作的需求開始被要求重拾Android; 都已經投向iOS陣營了..真是. 不多說, 重新邊學邊寫也是件好事. 這篇文章就來當作第一炮吧..哈!!!

3年前, 不知道Google是否已經有GSON這樣的東西(應該已經有了, 只是小弟太愚昧..呵), 要處理API返回的JSON, 都是透過以下方式

1
2
3
4
5
6
7
8
9
10
public TagData(JSONObject obj){
  try{
    this.serial = obj.getInt(ApiConsts.TAG_SERIAL);
    this.title = obj.getString(ApiConsts.TAG_TITLE);
    this.type = obj.getString(ApiConsts.TAG_TYPE);
    this.id = obj.optInt(ApiConsts.TAG_ID);
    this.num = obj.optInt(ApiConsts.TAG_NUMBERS);
    this.func = obj.getInt(ApiConsts.TAG_FUNC);
  }catch(Exception e){}
}

如果改用GSON後, Code就乾淨許多; 不用在那邊getter來 setter去

datasource.js
1
2
3
4
5
6
7
8
9
10
11
12
13
{"system":[
  {"tag":"6",
    "id":"6",
    "type":"pub",
    "tagT":"詐騙集團",
    "func":"0"},
  {"tag":"7",
    "id":"7",
    "type":"pub",
    "tagT":"信用卡業務",
    "func":"0"},...
  ]
}
TagObjMgr.java
1
2
3
public class TagObjMgr {
  public List<TagObj> system;
}

@SerializedName 這個annotation 就是你的JSON Object的key

TagObj.java
1
2
3
4
5
6
7
8
9
10
public class TagObj {
  @SerializedName("id")
  public int id;

  @SerializedName("tag")
  public int serial;

  @SerializedName("tagT")
  public String title;
}
MainActiviy.java
1
2
3
4
5
6
Gson gson = new Gson();
TagObjMgr response = gson.fromJson(jsonStr, TagObjMgr.class);
List<TagObj> objs = response.system;
for(TagObj obj: objs){
  Log.i(TAG, "======> " + obj.title);
}

既然Google都有這樣方便的工具了, 幹嘛造一堆輪子, 能偷懶就偷懶吧!

GSON

Too Many Open Files

| Comments

最近常遇到 Too many open files 的錯誤訊息, 不管是 memcached / nodejs .. 這跟Server上面的ulimit有關. 必須調整1024(預設)值.

顯示目前限制

1
$ulimit -a

編輯 /etc/pam.d/common-session, 加上

1
session required pam_limits.so

編輯 /etc/security/limits.conf

1
2
* soft nofile 51200
* hard nofile 51200

編輯 /etc/profile

1
ulimit -SHn 51200

執行

1
ulimit -SHn 51200