恒｜知識

2015年4月8日星期三

反組譯並修改 Android 應用程式實例

Reference: http://jserv.blogspot.hk/2010/05/android.html

為了某個實驗的動機，我們評估反編譯 Android 應用程式的可行性，本文即是筆者的心得與實際的範例，僅供參考。就筆者的認知，目前還沒有針對 Android 的 DEX to Java source 反編譯工具，可實際處理一般的 Android 應用程式，多半要繞幾圈，還會得到不甚理想的結果，不過，smali 這個反組譯工具，已是可用了，只是得對付類似 Jasmin 語法的 Dalvik 組合語言。筆者打包了 smali 與 Frozen Bubble for Android，作為示範：

http://0xlab.org/~jserv/dex-dis-example.tar.bz2

在 GNU/Linux 環境中，首先取得 Android SDK，這裡採用 Eclair/2.1，工具執行檔的路徑是 android-sdk-linux_86/tools，將此路徑放入 $PATH 環境變數，如此一來，就可以操作 adb, aapt, apkbuilder 一類的工具程式。筆者提供的套件已包含 Frozen Bubble 執行檔，檔名為 "FrozenBubble-orig.apk"。一旦 Android emulator 啟動後，即可安裝進去執行：(後續的操作也需要 Emulator 持續開啟)

$ adb install -r FrozenBubble-orig.apk

以下是執行畫面：

當然，沒必要讓筆者置喙談如何玩這個經典遊戲，不過我們倒是想更改原本的行為。在進行之前，我們先來複習 Android APK 的建立，參考 "How to build Android application package (.apk) from the command line using the SDK tools + continuously integrated using CruiseControl." 一文，我們可從以下圖表知悉細部的流程：

假設我們完全無法取得原始程式碼，該如何進行呢？沒錯，就透過 smali，簡化繁瑣的流程，筆者包裝為 Makefile，所以只要先解開並反組譯：

$ make extract

這時候會看到兩個目錄：

smali-src : 存放反組譯的程式檔輸出
workspace : 原本 Frozen Bubble 的 Android resource files

二話不說，當然是觀察反組譯的結果：

smali-src$ find
./org/jfedor/frozenbubble/FrozenBubble.smali
./org/jfedor/frozenbubble/R$id.smali
./org/jfedor/frozenbubble/GameView.smali
./org/jfedor/frozenbubble/SoundManager.smali
./org/jfedor/frozenbubble/LaunchBubbleSprite.smali
./org/jfedor/frozenbubble/Compressor.smali
./org/jfedor/frozenbubble/R$attr.smali
./org/jfedor/frozenbubble/BubbleFont.smali
./org/jfedor/frozenbubble/PenguinSprite.smali
./org/jfedor/frozenbubble/GameView$GameThread.smali
./org/jfedor/frozenbubble/BubbleSprite.smali./org/jfedor/frozenbubble/R$string.smali
./org/jfedor/frozenbubble/R$drawable.smali
./org/jfedor/frozenbubble/ImageSprite.smali
./org/jfedor/frozenbubble/BubbleManager.smali
./org/jfedor/frozenbubble/GameScreen.smali
./org/jfedor/frozenbubble/R.smali
./org/jfedor/frozenbubble/R$layout.smali
./org/jfedor/frozenbubble/BmpWrap.smali./org/jfedor/frozenbubble/FrozenGame.smali
./org/jfedor/frozenbubble/Sprite.smali
./org/jfedor/frozenbubble/LevelManager.smali
./org/jfedor/frozenbubble/R$raw.smali

就忠實地依據 Java package 的方式呈現，檔名結尾是 ".smali"。筆者的修改目標是，讓一開始的關卡 (Level) 從第一關直接跳躍到第五關。在 smali 原始程式碼 (注意：這完全不同於 Java 原始程式碼，而是極為貼近 Dalvik VM 所接受的 DEX 檔案的組合語言形式) 搜尋 "Level" 字眼，可發現主要的分佈就落於兩個檔案：

./org/jfedor/frozenbubble/GameView$GameThread.smali
./org/jfedor/frozenbubble/LevelManager.smali

前者就是 org/jfedor/frozenbubble/GameView.java 的編譯輸出，因為是 inner class，獨立編譯出 org/jfedor/frozenbubble/GameView.class$GameThread.smali，由這個 class 命名方式大概就可猜出其重要性，基本上只要能夠控制此 class 的實做，就掌握此 Android 應用程式的行為。而 class LevelManager 顧名思義，看來掌握了遊戲進行的程序控制。先觀察其 method 列表：

smali-src$ grep "\.method" org/jfedor/frozenbubble/LevelManager.smali
.method public constructor <init>([BI)V
.method private getLevel(Ljava/lang/String;)[[B
.method public getCurrentLevel()[[B
.method public getLevelIndex()I
.method public goToFirstLevel()V
.method public goToNextLevel()V.method public restoreState(Landroid/os/Bundle;)V
.method public saveState(Landroid/os/Bundle;)V

倘若我們以 "goToFirstLevel" 一類的關鍵字，在 org/jfedor/frozenbubble/GameView$GameThread.smali 檔案中搜尋，可找出有具體的呼叫行為：

smali-src$ grep -r goToFirstLevel *
org/jfedor/frozenbubble/GameView$GameThread.smali: invoke-virtual {v2}, Lorg/jfedor/frozenbubble/LevelManager;->goToFirstLevel()V
org/jfedor/frozenbubble/LevelManager.smali:.method public goToFirstLevel()V

由此更確定我們之前的猜想。其中組合語言寫為以下：

move-object/from16 v0, p0

iget-object v0, v0, Lorg/jfedor/frozenbubble/GameView$GameThread;->mLevelManager:Lorg/jfedor/frozenbubble/LevelManager;

move-object v2, v0

invoke-virtual {v2}, Lorg/jfedor/frozenbubble/LevelManager;->goToFirstLevel()V

不要被貌似複雜的語法嚇到了，基本上掌握 Java 程式語言的原則 "Everything is Object" (不過仍有提供 primitive type)，組合語言仍會作 Java Object 的實體化 (instantialization)，Dalvik 本身是 Register-based Virtual Machine，而扣除 static/class method 外，Java 中所有的 method invocation 多為 virtual function (對應於 C++ 的觀點，才能更具體用機械方式思考)，所以組合語言的指令為 "invoke-virtual" (注意有連字號，此與 Java bytecode 不同)，"{v2}" 表示第一受者的 Register，此為 Object 本體。接著，與 Java bytecode 一樣，"Lorg/jfedor/frozenbubble/LevelManager;" 就表示 Java 層面的 class "org.jfedor.frozenbubble.LevelManager"，字母 "L" 即為 class 的識別，而 "->" 就很直觀了，自然是 method invocation，所以連貫來看，這一段組合語言的 Java 意思為：

objectLevelManager.goToFirstLevel();

其中 objectLevelManager 是一個 class LevelManager 的實例/實體 (instance)。倘若需要在 method invocation 時，帶入參數，那麼前述的 "{v2}" 一般會被替換為 "{v0, v1, v2, ...} 的 register 列表。關於詳細的狀況，可參考 Dalvik 非官方說明，另外smali 的 wiki 也提供一些範例，可多加利用。

回到筆者剛剛設定的目標，我們既然知道 class org.jfedor.frozenbubble.GameView$GameThread 掌控了程式處理邏輯，自然一堆變數的傳遞、method 呼叫，都在此進行，那我們先試著用 "level" 字串去搜尋，想辦法找出常數定義，後者在 Dalvik 中，會集中保存於 constant pool 中，而 smali 的組合語言寫法大致是 "const" 開頭的宣告，端看其類型而定。以程式追蹤的目的來說，我們專注於以下兩種：

const-string : primitive string (不同於 java.lang.String) 表示
const/4 : 長度為 4 bytes (32 bit) 的整數表示

字串何其多，依據之前的推測來說，我們要找接近 "LevelManager" 字眼的組合語言程式碼，道理很明顯，從一般 Java programmer 的寫作邏輯去反推。經過一番搜尋，我們找到以下的反組譯程式碼： (位於檔案 org/jfedor/frozenbubble/GameView$GameThread.smali中)

const-string v3, "level"
const/4 v4, 0x0
move-object/from16 v0, v25 move-object v1, v3
move v2, v4
invoke-interface {v0, v1, v2}, Landroid/content/SharedPreferences;->getInt(Ljava/lang/String;I)I
move-result p4
new-instance v3, Lorg/jfedor/frozenbubble/LevelManager;
move-object v0, v3
move-object/from16 v1, v22
move/from16 v2, p4 invoke-direct {v0, v1, v2}, Lorg/jfedor/frozenbubble/LevelManager;-><init>([BI)V

在上述程式碼列表中，"Lorg/jfedor/frozenbubble/LevelManager;-><init>" 表示呼叫 class LevelManager 的 constructor，也就是 "<init>"。注意到 method invocation 方式就不同了，是 "invoke-direct"，表示 class constructor，而這之前要有 "new-instance v3, Lorg/jfedor/frozenbubble/LevelManager;" 的組合語言指令宣告。

前面談過「倘若需要在 method invocation 時，帶入參數，一般會被替換為 "{v0, v1, v2, ...} 的 register 列表」這樣的概念，我們可推知，Register v1 與 v2 就是實際上 class org.jfedor.frozenbubble.LevelManager 的 constructor 參數。就程式設計的邏輯來看，class GameView 就是依據某個流程，要求 LevelManager 去改變狀態，所以這裡的兩個參數，其實就是初始值，非常的重要。

與 Register v1 相關的組合語言指令有這幾行：(用粗體字標示)

const-string v3, "level"
const/4 v4, 0x0
move-object/from16 v0, v25
move-object v1, v3
move v2, v4
invoke-interface {v0, v1, v2}, Landroid/content/SharedPreferences;->getInt(Ljava/lang/String;I)I
move-result p4
new-instance v3, Lorg/jfedor/frozenbubble/LevelManager;
move-object v0, v3
move-object/from16 v1, v22
move/from16 v2, p4
invoke-direct {v0, v1, v2}, Lorg/jfedor/frozenbubble/LevelManager;-><init>([BI)V

顯然，Register v1 還被帶入到 android.content.Shared.Preference.getInt() method，而更早以前，其內含值被設定為 Register v3 的值，也就是常數字串 (const-string) "level"，這好像與我們的焦點不同。另外，像是 Register v22 這個編號較大的 register，表示 local variable，這點需要多留意，因為 Java 程式設計的規範來說，往往會將程式切割為若干 method，而 method 實做體中，又有極多的 local variable，於是往往可從組合語言反推 Java 原始碼的型態。

那麼，看看 Register v2 吧，同樣用粗體字標示相關的指令：

const-string v3, "level"
const/4 v4, 0x0
move-object/from16 v0, v25
move-object v1, v3
move v2, v4
invoke-interface {v0, v1, v2}, Landroid/content/SharedPreferences;->getInt(Ljava/lang/String;I)I
move-result p4
new-instance v3, Lorg/jfedor/frozenbubble/LevelManager;
move-object v0, v3
move-object/from16 v1, v22
move/from16 v2, p4
invoke-direct {v0, v1, v2}, Lorg/jfedor/frozenbubble/LevelManager;-><init>([BI)V

這個 Register v4 的內含值 "0x0" 會指派到 Register v2 中，而讓我們似乎找到方向了，回頭看看 class org.jfedor.frozenbubble.LevelManager 的 constructor 宣告方式： (之前 grep 結果的第一行)

smali-src$ grep "\.method" org/jfedor/frozenbubble/LevelManager.smali
.method public constructor <init>([BI)V

其中 "public" 是 ACL (存取權限) 的宣告，而 constructor 的符號規範為 "<init>"，注意到括號 "(" 與 ")" 裡面的兩個大寫字母，表示接受兩個參數，對應的型態為：

B : byte
I : int

在 Java 與 Dalvik virtual machine 皆以同樣的形式作宣告，看到這裡，我們實在忍不住要動手修改了，當然，一切都是實驗性質。既然一開啟遊戲，畫面就顯示 Level 1，表示 LevelManager 一開始接受的 level 參數會是 "0"，然後依據 GameView 的邏輯，逐一升級或中斷遊戲進行，那麼，如果要讓遊戲一起動就是 Level 5，是不是要把內含值改為 0x4 即可？也就是 "invoke-direct {v0, v1, v2}, Lorg/jfedor/frozenbubble/LevelManager;-><init>([BI)V" 的 Register v2 當時內含值該為 0x4，不就任務達成了嗎？想到此，不禁會心一笑。

不過，回顧稍早 Register v2 的相關程式碼輸出，其中有兩行需要留意 (以粗體字為主)：

invoke-interface {v0, v1, v2}, Landroid/content/SharedPreferences;->getInt(Ljava/lang/String;I)I
move-result p4
new-instance v3, Lorg/jfedor/frozenbubble/LevelManager;
move-object v0, v3
move-object/from16 v1, v22
move/from16 v2, p4

"p4" 用以保存 method invocation 之後的回傳值，顯然，Register v2 受到 p4 的指派，也就是被更動為 android.content.Shared.Preference.getInt() method 的回傳值，這存在不確定性，於是，我們乾脆一口氣改掉： (修改的部份會用井字號 "#" 作註解)

# Modified from 0x0 to 0x4"
const/4 v4, 0x4
move-object/from16 v0, v25
move-object v1, v3
move v2, v4
# Modified: removed the following 2 lines
# invoke-interface {v0, v1, v2}, Landroid/content/SharedPreferences;->getInt(Ljava/lang/String;I)I
# move-result p4
new-instance v3, Lorg/jfedor/frozenbubble/LevelManager;
move-object v0, v3
move-object/from16 v1, v22
# Modified: removed the following 1 line
# move/from16 v2, p4
invoke-direct {v0, v1, v2}, Lorg/jfedor/frozenbubble/LevelManager;-><init>([BI)V

改好程式，當然要驗證，回到上一層目錄，透過 smali 提供的組譯器，重新產生 Dalvik DEX 輸出，為了簡化流程，筆者把 smali, apkbuilder, aapt, adb install 都一次整合進去，所以會直接讓 Android Emulator 生效，來看看我們的戰果吧：

注意到左下角，這表示我們成功了，完全不用取得 Java 原始程式碼，就可以作反組譯並且修改的動作。

2015年3月29日星期日

Batch script to replace PHP short open tags with

find ./ -name "*.php" -type f | xargs perl -i.sed.bak -p -e 's|<\?=(.*?)\?>|<?php echo \1; ?>|g; s|<\?|<?php|g; s|<\?phpphp|<?php|g; s|<\?php([^ ])|<?php \1|g; s|([^ ])\?>|\1 ?>|g'

find ./ -name "*.sed.bak" -type f -delete

Set Up Web Server

apt-get install python-software-properties

Install Nginx Latest Version:

add-apt-repository ppa:nginx/stable
aptitude update
aptitude install nginx

Install PHP Latest Version:

add-apt-repository ppa:ondrej/php5
apt-get update

apt-get install php5-fpm

2015年3月27日星期五

開啟opcache導致502 bad gateway問題

本想開啟opcache來做code cache，木想到開了就502錯誤，googe許久也找不到解決方案，上stackoverflow提問也木有人回答。。。
如果不加載opcache.so 就一切正常，說明是opcache的內部問題。。。看nginx error.log 和php5-fpm.log也找不到什麼解決方法。。

我懷疑是版本兼容的問題，我使用的PHP版本是ubuntu apt源默認的版本，也就是php5.3.10-ubuntu ，因為在服務器上和本地2台機器都是這樣，一加載opcache訪問就報502錯誤。。。
中午趁有休息的時間，速度把PHP版本升級到5.5，這樣就可以使用內置的opcache（PHP5.5開始默認帶zend opcache，而且是默認開啟的）。。。

add-apt-repository ppa:ondrej/php5
apt-get update
apt-get install php5-fpm

按照上面代碼就可以把系統的php版本升級到最新的stable版本，目前是PHP5.5.5。。。
果然，安裝好後，一切正常。。。我看了下PHP官方，PHP5.3的穩定版本是PHP5.3.27，看來真的是APT源默認的版本和opcache 7.0.2不兼容。。。

stay up-to-date with PHP

You could use a PPA to stay up-to-date with PHP. I use:

https://launchpad.net/~ondrej/+archive/php5 Launchpad logo

It's now on 5.5 and also includes Apache 2.4 update. For Apache 2.2+ PHP 5.4 repository, see the bottom of the answer.

If you want use this PHP, do this:

ppa:ondrej/php5 Launchpad logo

(Click here for instructions on using PPAs.)

sudo add-apt-repository ppa:ondrej/php5
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install php5

If you don't have add-apt-repository binary do the following:

sudo apt-get install python-software-properties

Precautions:

Are PPA's safe to add to my system and what are some "red flags" to watch out for?

For Ubuntu 12.10

Ubuntu 12.10's official repository is updated to PHP 5.4. You should use the main repository package if you can. To do this, just install PHP5 Install php5

without enabling any PPA.

PHP 5.4

Prepackaged latest PHP 5.4 now resides in separate PPA: ppa:ondrej/php5-oldstable Launchpad logo

Configuring Your LEMP System (Linux, nginx, MySQL, PHP-FPM) For Maximum Performance

Version 1.0
Author: Falko Timme

Follow me on Twitter
Last edited 09/19/2012

If you are using nginx as your webserver, you are looking for a performance boost and better speed. nginx is fast by default, but you can optimize its performance and the performance of all parts (like PHP and MySQL) that work together with nginx. Here is a small, incomprehensive list of tips and tricks to configure your LEMP system (Linux, nginx, MySQL, PHP-FPM) for maximum performance. These tricks work for me, but your mileage may vary. Do not implement them all at once, but one by one and check what effect the modification has on your system's performance.

I do not issue any guarantee that this will work for you!

1 Reducing Disk I/O By Mounting PArtitions With noatime And nodiratime

Add noatime and nodiratime to your mount options in /etc/fstab:

vi /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda2 during installation
UUID=9cc886cd-98f3-435a-9830-46b316e2a20e /               ext3    errors=remount-ro,noatime,nodiratime,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0 0       1
# swap was on /dev/sda1 during installation
UUID=bba13162-121d-40a4-90a7-10f78a0097ae none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0

#Parallels Shared Folder mount
none         /media/psf   prl_fs   sync,nosuid,nodev,noatime,share,nofail     0       0

Remount the modified partitions as follows (make sure you use the correct mount point for each partition):

mount -o remount /

You can read more about this in this howto: Reducing Disk IO By Mounting Partitions With noatime

2 Tuning nginx

2.1 worker_processes

Make sure you use the correct amount of worker_processes in your /etc/nginx/nginx.conf. This should be equal to the amount of CPU cores in the output of

cat /proc/cpuinfo | grep processor

root@server1:~# cat /proc/cpuinfo | grep processor
processor : 0
processor : 1
processor : 2
processor : 3
processor : 4
processor : 5
processor : 6
processor : 7
root@server1:~#

In this example, we have eight CPU cores, so we set

vi /etc/nginx/nginx.conf

[...]
worker_processes 8;
[...]

2.2 keepalive_timeout, sendfile, tcp_nopush, tcp_nodelay

Set keepalive_timeout to a sensible value like two seconds. Enable sendfile, tcp_nopush, and tcp_nodelay:

vi /etc/nginx/nginx.conf

[...]
http {
[...]
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 2;
        types_hash_max_size 2048;
        server_tokens off;
[...]
}
[...]

2.3 File Cache

Enable the nginx file cache:

vi /etc/nginx/nginx.conf

[...]
http {
[...]
        ##
        # File Cache Settings
        ##

        open_file_cache          max=5000  inactive=20s;
        open_file_cache_valid    30s;
        open_file_cache_min_uses 2;
        open_file_cache_errors   on;
[...]
}
[...]

2.4 Enable Gzip Compression

You can read more about Gzip compression here: How To Save Traffic With nginx's HttpGzipModule (Debian Squeeze)

vi /etc/nginx/nginx.conf

[...]
http {
[...]
        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_static on;
        gzip_disable "msie6";
        gzip_http_version 1.1;
        gzip_vary on;
        gzip_comp_level 6;
        gzip_proxied any;
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript text/x-js;
        gzip_buffers 16 8k;
[...]
}
[...]

2.5 Enable The SSL Session Cache

If you serve https web sites, you should enable the SSL session cache:

vi /etc/nginx/nginx.conf

[...]
http {
[...]
        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  10m;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;
[...]
}
[...]

2.6 Use The FastCGI Cache

If you have cacheable PHP content, you can use the nginx FastCGI cache to cache that content. In your nginx.conf, add a line similar to this one:

vi /etc/nginx/nginx.conf

[...]
http {
[...]
        fastcgi_cache_path /var/cache/nginx levels=1:2 keys_zone=microcache:10m max_size=1000m inactive=60m;
[...]
}
[...]

The cache directory /var/cache/nginx must exist and be writable for nginx:

mkdir /var/cache/nginx
chown www-data:www-data /var/cache/nginx

(By using tmpfs, you can even place the directory directly in your server's memory which provides another small speed advantage - take a look at this tutorial to learn more: Storing Files/Directories In Memory With tmpfs).

In your vhost configuration, add the following block to your location ~ \.php$ {} section (you can modify it depending on when content should be cached and when not):

[...]
                # Setup var defaults
                set $no_cache "";
                # If non GET/HEAD, don't cache & mark user as uncacheable for 1 second via cookie
                if ($request_method !~ ^(GET|HEAD)$) {
                    set $no_cache "1";
                }
                # Drop no cache cookie if need be
                # (for some reason, add_header fails if included in prior if-block)
                if ($no_cache = "1") {
                    add_header Set-Cookie "_mcnc=1; Max-Age=2; Path=/";
                    add_header X-Microcachable "0";
                }
                # Bypass cache if no-cache cookie is set
                if ($http_cookie ~* "_mcnc") {
                            set $no_cache "1";
                }
                # Bypass cache if flag is set
                fastcgi_no_cache $no_cache;
                fastcgi_cache_bypass $no_cache;
                fastcgi_cache microcache;
                fastcgi_cache_key $scheme$host$request_uri$request_method;
                fastcgi_cache_valid 200 301 302 10m;
                fastcgi_cache_use_stale updating error timeout invalid_header http_500;
                fastcgi_pass_header Set-Cookie;
                fastcgi_pass_header Cookie;
                fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
[...]

So the full location ~ \.php$ {} section could look as follows:

[...]
location ~ \.php$ {

                # Setup var defaults
                set $no_cache "";
                # If non GET/HEAD, don't cache & mark user as uncacheable for 1 second via cookie
                if ($request_method !~ ^(GET|HEAD)$) {
                    set $no_cache "1";
                }
                # Drop no cache cookie if need be
                # (for some reason, add_header fails if included in prior if-block)
                if ($no_cache = "1") {
                    add_header Set-Cookie "_mcnc=1; Max-Age=2; Path=/";
                    add_header X-Microcachable "0";
                }
                # Bypass cache if no-cache cookie is set
                if ($http_cookie ~* "_mcnc") {
                            set $no_cache "1";
                }
                # Bypass cache if flag is set
                fastcgi_no_cache $no_cache;
                fastcgi_cache_bypass $no_cache;
                fastcgi_cache microcache;
                fastcgi_cache_key $scheme$host$request_uri$request_method;
                fastcgi_cache_valid 200 301 302 10m;
                fastcgi_cache_use_stale updating error timeout invalid_header http_500;
                fastcgi_pass_header Set-Cookie;
                fastcgi_pass_header Cookie;
                fastcgi_ignore_headers Cache-Control Expires Set-Cookie;

                try_files $uri =404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/var/lib/php5-fpm/web1.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param PATH_INFO $fastcgi_script_name;
                fastcgi_intercept_errors on;
}
[...]

This would cache pages with the return codes 200, 301, and 302 for ten minutes.

You can read more about this topic here: Why You Should Always Use Nginx With Microcaching

2.7 Use FastCGI Buffers

In your vhost configuration, you can add the following lines to your location ~ \.php$ {} section:

[...]
                fastcgi_buffer_size 128k;
                fastcgi_buffers 256 16k;
                fastcgi_busy_buffers_size 256k;
                fastcgi_temp_file_write_size 256k;
                fastcgi_read_timeout 240;
[...]

The full location ~ \.php$ {} section could look as follows:

[...]
location ~ \.php$ {
                try_files $uri =404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass unix:/var/lib/php5-fpm/web1.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param PATH_INFO $fastcgi_script_name;
                fastcgi_intercept_errors on;

                fastcgi_buffer_size 128k;
                fastcgi_buffers 256 16k;
                fastcgi_busy_buffers_size 256k;
                fastcgi_temp_file_write_size 256k;
                fastcgi_read_timeout 240;
}
[...]

2.8 Use memcached

nginx can read full pages directly from memcached. So if your web application is capable of storing full pages in memcached, nginx can fetch that page from memcached. An example configuration (in your vhost) would be as follows:

[...]
        location ~ \.php$ {
                set $no_cache "";
                if ($query_string ~ ".+") {
                        set $no_cache "1";
                }
                if ($request_method !~ ^(GET|HEAD)$ ) {
                        set $no_cache "1";
                }
                if ($request_uri ~ "nocache") {
                        set $no_cache "1";
                }
                if ($no_cache = "1") {
                        return 405;
                }

                set $memcached_key $host$request_uri;
                memcached_pass     127.0.0.1:11211;
                default_type text/html;
                error_page 404 405 502 = @php;
                expires epoch;
        }

        location @php {
                        try_files $uri =404;
                        include /etc/nginx/fastcgi_params;
                        fastcgi_pass unix:/var/lib/php5-fpm/web1.sock;
                        fastcgi_index index.php;
                        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                        fastcgi_param PATH_INFO $fastcgi_script_name;
                        fastcgi_intercept_errors on;
        }
[...]

It is important that your web application uses the same key for storing pages in memcached that nginx uses to fetch these pages from memcached (in this example it's $host$request_uri), otherwise this will not work.

If you store lots of data in memcached, make sure you have allocated enough RAM to memcached, e.g.:

vi /etc/memcached.conf

[...]
# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default
# Note that the daemon will grow to this size, but does not start out holding this much
# memory
-m 512
[...]

2.9 Make Browsers Cache Static Files With The expires Directive

Files (like images, CSS, JS, etc.) that don't change often can be cached by the visitor's browser by using the expires directive (seehttp://wiki.nginx.org/HttpHeadersModule#expires):

[...]
               location ~*  \.(jpg|jpeg|png|gif|ico)$ {
                         expires 365d;
               }
[...]

2.10 Disable Logging For Static Files

Normally it doesn't make much sense to log images or CSS files in the access log. To reduce disk I/O, we can disable logging for such files, e.g. as follows:

[...]
               location ~*  \.(jpg|jpeg|png|gif|ico)$ {
                         log_not_found off;
                         access_log off;
               }
[...]

3 Tuning PHP-FPM

3.1 Use A PHP Opcode Cache Like Xcache Or APC

Make sure you have a PHP opcode cache such as Xcache or APC installed. On Debian/Ubuntu, Xcache can be installed as follows:

apt-get install php5-xcache

APC can be installed as follows:

apt-get install php-apc

Make sure you have just one installed (either Xcache or APC), not both. Reload PHP-FPM after the installation:

/etc/init.d/php5-fpm reload

3.2 Allocate Enough Memory To Xcache/APC

If you have lots of PHP scripts, you should probably raise the memory that is allocated to Xcache or APC. For Xcache, you can do this in/etc/php5/conf.d/xcache.ini:

vi /etc/php5/conf.d/xcache.ini

[...]
xcache.size  =                512M
[...]

Likewise for APC:

vi /etc/php5/conf.d/apc.ini

[...]
apc.shm_size="512"
[...]

Reload PHP-FPM after your modification:

/etc/init.d/php5-fpm reload

3.3 PHP-FPM Emergency Settings

This is more of a reliability setting than a performance setting: PHP-FPM can restart itself if it stops working:

vi /etc/php5/fpm/php-fpm.conf

[...]
; If this number of child processes exit with SIGSEGV or SIGBUS within the time
; interval set by emergency_restart_interval then FPM will restart. A value
; of '0' means 'Off'.
; Default Value: 0
emergency_restart_threshold = 10

; Interval of time used by emergency_restart_interval to determine when
; a graceful restart will be initiated.  This can be useful to work around
; accidental corruptions in an accelerator's shared memory.
; Available Units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
emergency_restart_interval = 1m

; Time limit for child processes to wait for a reaction on signals from master.
; Available units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
process_control_timeout = 10s
[...]

3.4 For PHP >= 5.3.9: Use The ondemand Process Manager

If you use PHP >= 5.3.9, you can use the ondemand process manager in a PHP-FPM pool instead of static or dynamic, this will save you some RAM:

[...]
pm = ondemand
pm.max_children = 100
pm.process_idle_timeout = 5s
[...]

3.5 Use Unix Sockets Instead Of TCP Sockets

To reduce networking overhead, you should configure your pools to use Unix sockets instead of TCP:

[...]
;listen = 127.0.0.1:9000
listen = /var/lib/php5-fpm/www.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
[...]

If you change this, you must of course adjust the location ~ \.php$ {} section in your nginx vhost to use the socket (fastcgi_pass unix:/var/lib/php5-fpm/www.sock; instead of fastcgi_pass 127.0.0.1:9000;):

[...]
location ~ \.php$ {
                try_files $uri =404;
                include /etc/nginx/fastcgi_params;
                ##fastcgi_pass 127.0.0.1:9000;
                fastcgi_pass unix:/var/lib/php5-fpm/www.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param PATH_INFO $fastcgi_script_name;
                fastcgi_intercept_errors on;
}
[...]

3.6 Avoid 502 Bad Gateway Errors With Sockets On Busy Sites

If you use Unix sockets with PHP-FPM, you might encounter random 502 Bad Gateway errors with busy websites. To avoid this, we raise the max. number of allowed connections to a socket. Open /etc/sysctl.conf...

vi /etc/sysctl.conf

... and set:

[...]
net.core.somaxconn = 4096
[...]

Run

sysctl -p

afterwards for the change to take effect.

4 Tuning MySQL

4.1 Optimize Your my.cnf

You should use scripts such as mysqltuner.pl or tuning-primer.sh (or both) to find out which settings you should adjust in your my.cnf file. One of the most important variables is query_cache_size, and, if you use InnoDB tables, innodb_buffer_pool_size.

This is an example configuration from a test server with 16GB RAM, about 30 databases with 50% MyISAM tables and 50% InnoDB tables - this worked out quite well for database-driven test sites that were stressed with a benchmark tool (ab):

[...]
key_buffer = 256M

max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 100

table_open_cache = 16384
table_definition_cache = 8192

sort_buffer_size = 256K

read_buffer_size = 128K

read_rnd_buffer_size = 256K

myisam_sort_buffer_size = 64M
myisam_use_mmap = 1
thread_concurrency = 10
wait_timeout = 30

myisam-recover = BACKUP,FORCE

query_cache_limit = 10M
query_cache_size = 1024M
query_cache_type = 1

join_buffer_size = 4M

log_slow_queries        = /var/log/mysql/mysql-slow.log
long_query_time = 1

expire_logs_days        = 10
max_binlog_size         = 100M

innodb_buffer_pool_size = 2048M
innodb_log_file_size = 256M
innodb_log_buffer_size = 16M
innodb_flush_log_at_trx_commit = 0
innodb_thread_concurrency = 8
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_io_capacity = 50000
innodb_flush_method = O_DIRECT
innodb_file_per_table
innodb_additional_mem_pool_size = 256M
transaction-isolation = READ-COMMITTED

innodb_support_xa = 0
innodb_commit_concurrency = 8
innodb_old_blocks_time = 1000
[...]

Please note: If you need ACID compliance, you must set innodb_flush_log_at_trx_commit to 1. You can find out more about this onhttp://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit.

innodb_io_capacity should be set to high values only if you use MySQL on an SSD. If you use it on a normal hard drive, you better leave that line out.

4.2 Use An SSD

You can get a big performance boost by using MySQL with a solid state disk (SSD) as this reduces disk I/O a lot. The easiest way to do this is by mounting the /var/lib/mysql directory to an SSD.

5 Web Application Caching

Lots of web applications (such as WordPress with the WP Super Cache or W3 Total Cache plugins, Drupal with the Boost module, TYPO3 with the nc_staticfilecache extension) offer the possibility to create a full page cache which is stored on the hard drive and which can be accessed directly by nginx so that it can bypass the whole PHP-MySQL stack. This provides a huge performance boost.

You can find tutorials about this here:

You can speed the static file cache up even more by placing it directly in the server's memory with the tmpfs filesystem:

Storing Files/Directories In Memory With tmpfs

Of course, you can use tmpfs also for the nginx FastCGI cache from chapter 2.6.

6 Links

訂閱：文章 (Atom)

2015年4月8日 星期三

反組譯並修改 Android 應用程式實例

2015年3月29日 星期日

Batch script to replace PHP short open tags with

Set Up Web Server

2015年3月27日 星期五

開啟opcache導致502 bad gateway問題

stay up-to-date with PHP

For Ubuntu 12.10

PHP 5.4

Configuring Your LEMP System (Linux, nginx, MySQL, PHP-FPM) For Maximum Performance

Configuring Your LEMP System (Linux, nginx, MySQL, PHP-FPM) For Maximum Performance

1 Reducing Disk I/O By Mounting PArtitions With noatime And nodiratime

2 Tuning nginx

2.1 worker_processes

2.2 keepalive_timeout, sendfile, tcp_nopush, tcp_nodelay

2.3 File Cache

2.4 Enable Gzip Compression

2.5 Enable The SSL Session Cache

2.6 Use The FastCGI Cache

2.7 Use FastCGI Buffers

2.8 Use memcached

2.9 Make Browsers Cache Static Files With The expires Directive

2.10 Disable Logging For Static Files

3 Tuning PHP-FPM

3.1 Use A PHP Opcode Cache Like Xcache Or APC

3.2 Allocate Enough Memory To Xcache/APC

3.3 PHP-FPM Emergency Settings

3.4 For PHP >= 5.3.9: Use The ondemand Process Manager

3.5 Use Unix Sockets Instead Of TCP Sockets

3.6 Avoid 502 Bad Gateway Errors With Sockets On Busy Sites

4 Tuning MySQL

4.1 Optimize Your my.cnf

4.2 Use An SSD

5 Web Application Caching

6 Links

2015年4月8日星期三

2015年3月29日星期日

2015年3月27日星期五