Skip to content

Commit e486a57

Browse files
committed
Merge remote-tracking branch 'origin/master' into delay_strategy
2 parents 01bcb21 + 2090835 commit e486a57

64 files changed

Lines changed: 3800 additions & 35 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Features:
1010
* [Feature rich](docs/quick_tour.md).
1111
* Implements [JMS](https://docs.oracle.com/javaee/7/api/javax/jms/package-summary.html) like transports based on a[queue-interop](https://github.com/queue-interop/queue-interop) interfaces.
1212
* Supported transports
13-
* AMQP based on [the ext](docs/transport/amqp.md), [the lib](docs/transport/amqp_lib.md)
13+
* Amqp based on [the ext](docs/transport/amqp.md), [bunny](docs/transport/amqp_bunny.md), [the lib](docs/transport/amqp_lib.md)
1414
* [Beanstalk](docs/transport/pheanstalk.md)
1515
* [STOMP](docs/transport/stomp.md)
1616
* [Amazon SQS](docs/transport/sqs.md)

‎bin/dev‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ while getopts "bustefcd" OPTION; do
2727
COMPOSE_PROJECT_NAME=mqdev docker-compose run -e CHANGELOG_GITHUB_TOKEN=${CHANGELOG_GITHUB_TOKEN:-""} --workdir="/mqdev" --rm generate-changelog github_changelog_generator --future-release "$2" --simple-list
2828
;;
2929

30-
d) COMPOSE_PROJECT_NAME=mqdev docker-compose run --workdir="/mqdev" --rm dev php pkg/enqueue-bundle/Tests/Functional/app/console.php config:dump-reference enqueue
30+
d) COMPOSE_PROJECT_NAME=mqdev docker-compose run --workdir="/mqdev" --rm dev php pkg/enqueue-bundle/Tests/Functional/app/console.php config:dump-reference enqueue -vvv
3131
;;
3232
\?)
3333
echo "Invalid option: -$OPTARG" >&2

‎bin/release‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ fi
1414

1515
CURRENT_BRANCH=`git rev-parse --abbrev-ref HEAD`
1616

17-
for REMOTE in origin stomp amqp-ext amqp-lib pheanstalk gearman sqs fs redis dbal null rdkafka enqueue simple-client enqueue-bundle job-queue test async-event-dispatcher
17+
for REMOTE in origin stomp amqp-ext amqp-lib amqp-bunny pheanstalk gearman sqs fs redis dbal null rdkafka enqueue simple-client enqueue-bundle job-queue test async-event-dispatcher
1818
do
1919
echo ""
2020
echo ""

‎bin/subtree-split‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ remote simple-client git@github.com:php-enqueue/simple-client.git
4848
remote stomp git@github.com:php-enqueue/stomp.git
4949
remote amqp-ext git@github.com:php-enqueue/amqp-ext.git
5050
remote amqp-lib git@github.com:php-enqueue/amqp-lib.git
51+
remote amqp-bunny git@github.com:php-enqueue/amqp-bunny.git
5152
remote pheanstalk git@github.com:php-enqueue/pheanstalk.git
5253
remote gearman git@github.com:php-enqueue/gearman.git
5354
remote fs git@github.com:php-enqueue/fs.git
@@ -66,6 +67,7 @@ split 'pkg/simple-client' simple-client
6667
split 'pkg/stomp' stomp
6768
split 'pkg/amqp-ext' amqp-ext
6869
split 'pkg/amqp-lib' amqp-lib
70+
split 'pkg/amqp-bunny' amqp-bunny
6971
split 'pkg/pheanstalk' pheanstalk
7072
split 'pkg/gearman' gearman
7173
split 'pkg/rdkafka' rdkafka

‎composer.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
"enqueue/stomp": "*@dev",
99
"enqueue/amqp-ext": "*@dev",
1010
"enqueue/amqp-lib": "*@dev",
11+
"enqueue/amqp-bunny": "*@dev",
1112
"enqueue/amqp-tools": "*@dev",
1213
"php-amqplib/php-amqplib": "^2.7@dev",
1314
"enqueue/redis": "*@dev",
@@ -80,6 +81,10 @@
8081
"type": "path",
8182
"url": "pkg/amqp-lib"
8283
},
84+
{
85+
"type": "path",
86+
"url": "pkg/amqp-bunny"
87+
},
8388
{
8489
"type": "path",
8590
"url": "pkg/amqp-tools"

‎docs/bundle/config_reference.md‎

Lines changed: 135 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,9 @@ enqueue:
6868
write_timeout: ~
6969
persisted: false
7070
lazy: true
71+
72+
# The receive strategy to be used. We suggest to use basic_consume as it is more performant. Though you need AMQP extension 1.9.1 or higher
73+
receive_method: basic_get # One of "basic_get"; "basic_consume"
7174
rabbitmq_amqp:
7275

7376
# The connection to AMQP broker set as a string. Other parameters are ignored if set
@@ -99,6 +102,86 @@ enqueue:
99102
persisted: false
100103
lazy: true
101104

105+
# The receive strategy to be used. We suggest to use basic_consume as it is more performant. Though you need AMQP extension 1.9.1 or higher
106+
receive_method: basic_get # One of "basic_get"; "basic_consume"
107+
108+
# The option tells whether RabbitMQ broker has delay plugin installed or not
109+
delay_plugin_installed: false
110+
amqp_lib:
111+
112+
# The connection to AMQP broker set as a string. Other parameters are ignored if set
113+
dsn: ~
114+
115+
# The host to connect too. Note: Max 1024 characters
116+
host: localhost
117+
118+
# Port on the host.
119+
port: 5672
120+
121+
# The user name to use. Note: Max 128 characters.
122+
user: guest
123+
124+
# Password. Note: Max 128 characters.
125+
pass: guest
126+
127+
# The virtual host on the host. Note: Max 128 characters.
128+
vhost: /
129+
130+
# Connection timeout. Note: 0 or greater seconds. May be fractional.
131+
connection_timeout: !!float 3
132+
read_write_timeout: !!float 3
133+
134+
# Timeout in for income activity. Note: 0 or greater seconds. May be fractional.
135+
read_timeout: 3
136+
137+
# Timeout in for outcome activity. Note: 0 or greater seconds. May be fractional.
138+
write_timeout: 3
139+
lazy: true
140+
stream: true
141+
insist: false
142+
keepalive: false
143+
144+
# The receive strategy to be used. We suggest to use basic_consume as it is more performant. Though you need AMQP extension 1.9.1 or higher
145+
receive_method: basic_get # One of "basic_get"; "basic_consume"
146+
heartbeat: 0
147+
rabbitmq_amqp_lib:
148+
149+
# The connection to AMQP broker set as a string. Other parameters are ignored if set
150+
dsn: ~
151+
152+
# The host to connect too. Note: Max 1024 characters
153+
host: localhost
154+
155+
# Port on the host.
156+
port: 5672
157+
158+
# The user name to use. Note: Max 128 characters.
159+
user: guest
160+
161+
# Password. Note: Max 128 characters.
162+
pass: guest
163+
164+
# The virtual host on the host. Note: Max 128 characters.
165+
vhost: /
166+
167+
# Connection timeout. Note: 0 or greater seconds. May be fractional.
168+
connection_timeout: !!float 3
169+
read_write_timeout: !!float 3
170+
171+
# Timeout in for income activity. Note: 0 or greater seconds. May be fractional.
172+
read_timeout: 3
173+
174+
# Timeout in for outcome activity. Note: 0 or greater seconds. May be fractional.
175+
write_timeout: 3
176+
lazy: true
177+
stream: true
178+
insist: false
179+
keepalive: false
180+
181+
# The receive strategy to be used. We suggest to use basic_consume as it is more performant. Though you need AMQP extension 1.9.1 or higher
182+
receive_method: basic_get # One of "basic_get"; "basic_consume"
183+
heartbeat: 0
184+
102185
# The option tells whether RabbitMQ broker has delay plugin installed or not
103186
delay_plugin_installed: false
104187
fs:
@@ -155,19 +238,69 @@ enqueue:
155238

156239
# the connection will be performed as later as possible, if the option set to true
157240
lazy: true
241+
amqp_bunny:
242+
243+
# The connection to AMQP broker set as a string. Other parameters are ignored if set
244+
dsn: ~
245+
246+
# The host to connect too. Note: Max 1024 characters
247+
host: localhost
248+
249+
# Port on the host.
250+
port: 5672
251+
252+
# The user name to use. Note: Max 128 characters.
253+
user: guest
254+
255+
# Password. Note: Max 128 characters.
256+
pass: guest
257+
258+
# The virtual host on the host. Note: Max 128 characters.
259+
vhost: /
260+
lazy: true
261+
262+
# The receive strategy to be used. We suggest to use basic_consume as it is more performant. Though you need AMQP extension 1.9.1 or higher
263+
receive_method: basic_get # One of "basic_get"; "basic_consume"
264+
heartbeat: 0
265+
rabbitmq_amqp_bunny:
266+
267+
# The connection to AMQP broker set as a string. Other parameters are ignored if set
268+
dsn: ~
269+
270+
# The host to connect too. Note: Max 1024 characters
271+
host: localhost
272+
273+
# Port on the host.
274+
port: 5672
275+
276+
# The user name to use. Note: Max 128 characters.
277+
user: guest
278+
279+
# Password. Note: Max 128 characters.
280+
pass: guest
281+
282+
# The virtual host on the host. Note: Max 128 characters.
283+
vhost: /
284+
lazy: true
285+
286+
# The receive strategy to be used. We suggest to use basic_consume as it is more performant. Though you need AMQP extension 1.9.1 or higher
287+
receive_method: basic_get # One of "basic_get"; "basic_consume"
288+
heartbeat: 0
289+
290+
# The option tells whether RabbitMQ broker has delay plugin installed or not
291+
delay_plugin_installed: false
158292
client:
159293
traceable_producer: false
160294
prefix: enqueue
161295
app_name: app
162-
router_topic: router
296+
router_topic: default
163297
router_queue: default
164298
router_processor: enqueue.client.router_processor
165299
default_processor_queue: default
166300
redelivered_delay_time: 0
167301
job: false
168302
async_events:
169303
enabled: false
170-
spool_producer: false
171304
extensions:
172305
doctrine_ping_connection_extension: false
173306
doctrine_clear_identity_map_extension: false
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Enqueue Elastica Bundle
2+
3+
Improves performance of `fos:elastica:populate` commands by distributing the work among consumers.
4+
The performance gain depends on how much consumers you run.
5+
For example 10 consumers may give you 5 to 7 times better performance.
6+
7+
## Installation
8+
9+
Install packages using [composer](https://getcomposer.org/)
10+
11+
```bash
12+
$ composer require enqueue/elastica-bundle friendsofsymfony/elastica-bundle
13+
```
14+
15+
Add bundles to `AppKernel`
16+
17+
```php
18+
<?php
19+
use Symfony\Component\HttpKernel\Kernel;
20+
21+
class AppKernel extends Kernel
22+
{
23+
public function registerBundles()
24+
{
25+
$bundles = [
26+
// ...
27+
28+
new Enqueue\Bundle\EnqueueBundle(),
29+
new Enqueue\ElasticaBundle\EnqueueElasticaBundle(),
30+
new FOS\ElasticaBundle\FOSElasticaBundle(),
31+
];
32+
33+
return $bundles;
34+
}
35+
}
36+
```
37+
38+
Here's an example of what your `FOSElasticaBundle` configuration may look like:
39+
40+
```yaml
41+
fos_elastica:
42+
clients:
43+
default: { host: %elasticsearch_host%, port: %elasticsearch_port% }
44+
indexes:
45+
app:
46+
index_name: app_%kernel.environment%
47+
types:
48+
blog:
49+
mappings:
50+
text: ~
51+
persistence:
52+
driver: orm
53+
model: AppBundle\Entity\Blog
54+
provider: ~
55+
listener: ~
56+
finder: ~
57+
```
58+
59+
Here's an example of what your EnqueueBundle configuration may look like:
60+
61+
```yaml
62+
enqueue:
63+
transport:
64+
default: 'file://%kernel.root_dir%/../var/messages'
65+
```
66+
67+
Sure you can configure other transports like: [rabbitmq, amqp, stomp and so on](https://github.com/php-enqueue/enqueue-dev/blob/master/docs/bundle/config_reference.md)
68+
Create a `fos_elastica_populate` queue on broker side, if needed.
69+
70+
## Usage
71+
72+
The bundle once registered automatically replaces Doctrine ORM provider by async one.
73+
So you have to run as usual
74+
75+
```bash
76+
$ ./bin/console fos:elastica:populate
77+
```
78+
79+
If you want to disable this behavior you can un register the bundle or use env var
80+
81+
```bash
82+
$ ENQUEUE_ELASTICA_DISABLE_ASYNC=1 ./bin/console fos:elastica:populate
83+
```
84+
85+
Run some consumers either using client (you might have to enable it) consume command:
86+
87+
```bash
88+
$ ./bin/console enqueue:consume --setup-broker -vvv
89+
```
90+
91+
or a transport one:
92+
93+
```bash
94+
$ ./bin/console enqueue:transport:consume enqueue_elastica.populate_processor -vvv
95+
```
96+
97+
We suggest to use [supervisor](http://supervisord.org/) on production to control numbers of consumers and restart them.
98+
99+
Here's config example
100+
101+
```
102+
# cat /etc/supervisor/conf.d/fos_elastica_populate.conf
103+
[program:fos_elastica_populate]
104+
command=/mqs/symfony/bin/console enqueue:transport:consume fos_elastica_populate enqueue_elastica.populate_processor
105+
process_name=%(program_name)s_%(process_num)02d
106+
numprocs=10
107+
autostart=true
108+
autorestart=true
109+
startsecs=0
110+
user=root
111+
redirect_stderr=true
112+
```
113+
114+
[back to index](../index.md)

0 commit comments

Comments
 (0)