PHP CURL并发请求测试

现在有一个项目需求，需要对大量的用户请求进行回调。我需要用定时任务自动执行。

例如：某一时刻需要对100个用户进行回调时，如果使用循环依次执行curl，执行的时间可能会很长，因为我不知道curl请求的用户响应时间是多少，我需要控制整个脚本的执行时间以满足的定时任务的周期调用，同时需要尽可能满足每个curl请求的等待时间。

循环执行curl，由于PHP程序是串行的，只有等待上一个curl执行完毕后，下一个curl才会开始执行，而上一个curl的结束时间并不是十分确定，假设每个curl设置超时为30s，如果用户的服务在30s内做出响应则curl会提前结束，如果用户未能在30秒内做出响应，则最终curl将会执行30s。假设极端情况下，每个请求都占用30s，那么100个请求至少需要占用3000秒。如果在这未来的3000秒又产生300个请求等待回调，我们的服务将越来越难及时向用户发起回调。

为了提高回调效率，决定测试一下php的multi_curl这个功能。网上关于multi_curl的文档介绍比较少，不过国外早已经有程序员把它封装成了简单易用的模块，现在我就来测一下这个模块的性能是否满足我的需求。

GitHub项目 jmathai/php-multi-curl

安装方法

可以参考github readme.md，

composer require jmathai/php-multi-curl:dev-master -v

用例

Basic usage can be done using the addUrl($url/*, $options*/) method. This calls GET $url by passing in $options as the parameters.

<?php
  // Include Composer's autoload file if not already included.
  require '../vendor/autoload.php';

  // Instantiate the MultiCurl class.
  $mc = JMathai\PhpMultiCurl\MultiCurl::getInstance();

  // Make a call to a URL.
  $call1 = $mc->addUrl('http://slowapi.herokuapp.com/delay/2.0');
  // Make another call to a URL.
  $call2 = $mc->addUrl('http://slowapi.herokuapp.com/delay/1.0');

  // Access the response for $call2.
  // This blocks until $call2 is complete without waiting for $call1
  echo "Call 2: {$call2->response}\n";

  // Access the response for $call1.
  echo "Call 1: {$call1->response}\n";

  // Output a call sequence diagram to see how the parallel calls performed.
  echo $mc->getSequence()->renderAscii();

This is what the output of that code will look like.

Call 2: consequatur id est
Call 1: in maiores et
(http://slowapi.herokuapp.com/delay/2.0 ::  code=200, start=1447701285.5536, end=1447701287.9512, total=2.397534)
[====================================================================================================]
(http://slowapi.herokuapp.com/delay/1.0 ::  code=200, start=1447701285.5539, end=1447701287.0871, total=1.532997)
[================================================================                                    ]

高级用例

You'll most likely want to configure your cURL calls for your specific purpose. This includes setting the call's HTTP method, parameters, headers and more. You can use the addCurl($ch) method and configuring your curl handle using any of PHP's curl_* functions. 

<?php
  $mc = JMathai\PhpMultiCurl\MultiCurl::getInstance();

  // Set up your cURL handle(s).
  $ch = curl_init('http://www.google.com');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_POST, 1);

  // Add your cURL calls and begin non-blocking execution.
  $call = $mc->addCurl($ch);

  // Access response(s) from your cURL calls.
  $code = $call->code;

  • addUrl

    addUrl((string) $url/*, (array) $options*/)

    Makes a GET call to $url by passing the key/value array $options as parameters. This method automatically sets CURLOPT_RETURNTRANSFER to 1 internally.

    $call = $mc->addUrl('https://www.google.com', array('q' => 'github'));
echo $call->response;

  • addCurl

    addCurl((curl handle) $ch)

    Takes a curl handle $ch and executes it. This method, unlike addUrl, will not add anything to the cURL handle. You'll most likely want to set CURLOPT_RETURNTRANSFER yourself before passing the handle into addCurl.

    $ch = curl_init('http://www.mocky.io/v2/5185415ba171ea3a00704eed');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');

$call = $mc->addCurl($ch);
echo $call->response;

获取响应结果

The curl calls begin executing the moment you call addUrl or addCurl. Execution control is returned to your code immediately and blocking for the response does not occur until you access the response or code variables. The library only blocks for the call you're trying to access the response to and will allow longer running calls to continue to execute while returning control back to your code.

通过response 变量，用于获取响应的结果

echo $call->response;

通过code变量，获于HTTP response code

echo $call->code;

通过headers变量，获取响应的header信息，headers是个数组

var_dump($call->headers);

通过renderAscii方法，获取请求执行的时间记录

Return a string that prints out details of call latency and degree of being called in parallel. This method can be called indirectly through the multi-curl instance you're using.

echo $mc->getSequence()->renderAscii();

我主要记录一下我安装好后的一些测试结果：

我的测试代码：

// 这是发起请求的测试代码
public function multiCurl()
    {
        $mc = \JMathai\PhpMultiCurl\MultiCurl::getInstance();

        $urls = [
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 3],
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 2],
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 5],
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 4],
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 7],
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 2],
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 1],
            ['url' => 'http://blog.ifsvc.cn/test-curl', 'sleep' => 0],
        ];

        $ch = [];
        $results = [];
        foreach ($urls as $k => $item) {
            $ch[$k] = curl_init($item['url']);
            $header = [
                "HTTP_X_REAL_IP: " . REMOTE_ADDR,
                "appid: demo-test",
                "Content-Type: application/x-www-form-urlencoded"
            ];
            curl_setopt($ch[$k], CURLOPT_HTTPHEADER, $header);
            curl_setopt($ch[$k], CURLOPT_TIMEOUT, 15);
            curl_setopt($ch[$k], CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch[$k], CURLOPT_ENCODING, 'UTF-8');
            curl_setopt($ch[$k], CURLOPT_VERBOSE, 1);
            curl_setopt($ch[$k], CURLOPT_SSL_VERIFYPEER, FALSE);
            curl_setopt($ch[$k], CURLOPT_SSL_VERIFYHOST, FALSE);
            curl_setopt($ch[$k], CURLOPT_POST, TRUE);
            curl_setopt($ch[$k], CURLOPT_POSTFIELDS, http_build_query(['sleep' => $item['sleep']]));
            curl_setopt($ch[$k], CURLINFO_HEADER_OUT, TRUE);
            try {
                $call = $mc->addCurl($ch[$k]);
                $results[$k] = $call;
            } catch (Exception $ex) {
                echo "exception:" . $ex->getMessage();
            }
        }
        foreach ($results as $res) {
            echo $res->code . ' - ' . $res->response;
            echo "<br>";
            file_put_contents('./curl.log', $res->code . '-' . $res->response . PHP_EOL, FILE_APPEND);
        }
        echo $mc->getSequence()->renderAscii();
    }

http://blog.ifsvc.cn是我一个本地测试地址IP：127.0.0.1, test-curl接收请求的方法如下：

public function curl()
    {
        $time = $_POST['sleep'] ?? 0;
        sleep($time);
        echo "sleep-" . $time;
    }

即，接收到一个POST请求，请求参数为sleep时间，根据请求提交的sleep，程序等待一段时间再响应给请求端。

我把响应的结果记录到一个curl.log文件中：

200-sleep-3
200-sleep-2
200-sleep-5
200-sleep-4
200-sleep-7
200-sleep-2
200-sleep-1
200-sleep-0

浏览器输出的执行时间:

200 - sleep-3
200 - sleep-2
200 - sleep-5
200 - sleep-4
200 - sleep-7
200 - sleep-2
200 - sleep-1
200 - sleep-0
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3246, end=1563851180.3357, total=3.010959) [================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3248, end=1563851179.3356, total=2.010706) [======================= ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3249, end=1563851182.5743, total=5.249378) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.325, end=1563851182.5733, total=5.248249) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851186.3399, total=9.014747) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851182.3411, total=5.015868) [======================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3253, end=1563851183.3453, total=6.019901) [=================================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3254, end=1563851182.578, total=5.25248) [=========================================================== ]

可以看出，所有请求都是在 

200 - sleep-3
200 - sleep-2
200 - sleep-5
200 - sleep-4
200 - sleep-7
200 - sleep-2
200 - sleep-1
200 - sleep-0
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3246, end=1563851180.3357, total=3.010959) [================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3248, end=1563851179.3356, total=2.010706) [======================= ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3249, end=1563851182.5743, total=5.249378) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.325, end=1563851182.5733, total=5.248249) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851186.3399, total=9.014747) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851182.3411, total=5.015868) [======================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3253, end=1563851183.3453, total=6.019901) [=================================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3254, end=1563851182.578, total=5.25248) [=========================================================== ]

可以看出，所有请求都是在 

200 - sleep-3
200 - sleep-2
200 - sleep-5
200 - sleep-4
200 - sleep-7
200 - sleep-2
200 - sleep-1
200 - sleep-0
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3246, end=1563851180.3357, total=3.010959) [================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3248, end=1563851179.3356, total=2.010706) [======================= ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3249, end=1563851182.5743, total=5.249378) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.325, end=1563851182.5733, total=5.248249) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851186.3399, total=9.014747) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851182.3411, total=5.015868) [======================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3253, end=1563851183.3453, total=6.019901) [=================================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3254, end=1563851182.578, total=5.25248) [=========================================================== ]

可以看出，所有请求都是在 

200 - sleep-3
200 - sleep-2
200 - sleep-5
200 - sleep-4
200 - sleep-7
200 - sleep-2
200 - sleep-1
200 - sleep-0
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3246, end=1563851180.3357, total=3.010959) [================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3248, end=1563851179.3356, total=2.010706) [======================= ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3249, end=1563851182.5743, total=5.249378) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.325, end=1563851182.5733, total=5.248249) [=========================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851186.3399, total=9.014747) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3251, end=1563851182.3411, total=5.015868) [======================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3253, end=1563851183.3453, total=6.019901) [=================================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851177.3254, end=1563851182.578, total=5.25248) [=========================================================== ]

可以看出，所有请求都是在 1563851177 同一秒发出的，所有请求执行完毕，流览器的调试器中显示网络请求响应时间是9.02s。但是，我上面设置的最长延时是7s。

再看一下上面的记录，发现从第五个请求开始，响应时间就比预期的慢了不少；第5个请求设置的sleep 7s,而实际响应时间是9.01s； 第6个请求设置的是sleep 2s，实际响应时间却是5.01s，第8个请求设置的 sleep 1s, 实际响应时间是 6.01s; 最后一个请求sleep 0s，实际响应时间是5.25s。

PHP的并发到后面响应时间变慢的原因，我猜是由于PHP处理请求的worker数量导致的，找一下php-fpm的配置文件，在php-fpm.d/www.conf中找到以下配置参数

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes when pm is set to 'dynamic' or 'ondemand'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don't
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
; Note: This value is mandatory.
pm.max_children = 5

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 2

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 1

把上面的pm.max_children = 5 改成 10。然后重启一下php-fpm

再访问试一下：

200 - sleep-3
200 - sleep-2
200 - sleep-5
200 - sleep-4
200 - sleep-7
200 - sleep-2
200 - sleep-1
200 - sleep-0
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.6269, end=1563851980.1373, total=5.510375) [========================================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.6272, end=1563851979.1369, total=4.509555) [============================================================= ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.6273, end=1563851982.1385, total=7.51101) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.6275, end=1563851982.1357, total=7.508174) [====================================================================================================]
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.6276, end=1563851981.6903, total=7.062679) [=============================================================================================== ]
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.6277, end=1563851977.1329, total=2.504916) [================================== ]
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.6278, end=1563851977.133, total=2.504974) [================================== ]
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563851974.628, end=1563851977.1353, total=2.507141) [================================== ]

这次浏览器请求的，网络响应时间是7.56s，无论是从整体响应时间还是后面几个请求的响应时间来看，都有较大的改善。

接下来需要测试一下超时的情况，我把curl的timeout 设置为 5s，看一下浏览器的输出结果

200 - sleep-3
200 - sleep-2
0 - 
0 - 
0 - 
200 - sleep-2
200 - sleep-1
200 - sleep-0
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563860527.6287, end=1563860531.2911, total=3.662377) [========================================================================== ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563860527.6288, end=1563860530.6222, total=2.993253) [============================================================ ] 
(http://blog.ifsvc.cn/test-curl :: code=0, start=1563860527.6289, end=1563860532.6281, total=4.999043) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=0, start=1563860527.629, end=1563860532.6285, total=4.999378) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=0, start=1563860527.6291, end=1563860532.6286, total=4.999308) [====================================================================================================] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563860527.6292, end=1563860531.6225, total=3.993225) [================================================================================ ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563860527.6293, end=1563860531.6288, total=3.999353) [================================================================================= ] 
(http://blog.ifsvc.cn/test-curl :: code=200, start=1563860527.6295, end=1563860530.6251, total=2.995558) [============================================================ ]

可以看到有3个请求时接接近5s，因为超时直接结算了没有返回结果，获取到的HTTP response code=0. 网络请求时间为 5.56s。