|
|
# Purpose of these scripts/scenarios
|
|
|
# Purpose of the scripts/scenarios
|
|
|
|
|
|
Validate _multiple_ *publishers* (aka. producers) and _multiple_ *consumers* working on the same queue, exchange messages via some AMQP message broker without any message corruption. Messages consumed should only be consumed *once*; In other words, no consumer should receive a message that has already been seen by another consumer (it's in question if this will by technically proven).
|
|
|
Validate _multiple_ *publishers* (aka. producers) and _multiple_ *consumers* working on the same [queue](https://en.wikipedia.org/wiki/Message_queue), exchange messages via some [AMQP](https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol) [message broker](https://en.wikipedia.org/wiki/Message_broker) without any message [corruption](https://en.wikipedia.org/wiki/Data_corruption). Messages consumed should only be consumed *once*; In other words, no consumer should receive a message that has already been seen by another consumer.
|
|
|
|
|
|
## Scenario 1
|
|
|
|
|
|
* Scope: Check valid message exchange, [cryptographically](https://en.wikipedia.org/wiki/Cryptography) proven message exchange
|
|
|
* Out of scope: [Performance](https://en.wikipedia.org/wiki/Performance_indicator) data
|
|
|
* Algorithm used to prove correct message exchange: Similar to many blockchains, a (simplistic) [PoW](https://en.wikipedia.org/wiki/Proof-of-work_system)
|
|
|
* *Scope*: Check valid message exchange, [cryptographically](https://en.wikipedia.org/wiki/Cryptography) proven message exchange
|
|
|
* *Out of scope*:
|
|
|
- [Performance](https://en.wikipedia.org/wiki/Performance_indicator) data
|
|
|
- Validate if messages are only consumed once (since this is a requirement of MQ)
|
|
|
* *Algorithm* used to prove correct message exchange: Similar to many [blockchains](https://en.wikipedia.org/wiki/Blockchain), a (simplistic) [PoW](https://en.wikipedia.org/wiki/Proof-of-work_system)
|
|
|
|
|
|
### Producer
|
|
|
|
|
|
Generate hash over the last produced PoW, last produced proof and new proof, where proof is a prime number (to further increase complexity and unpredictability, this is a random number (between fixed MIN/MAX, however, the resulting prime can be larger than MAX) and the hash starts with n number of 0s (zeros).
|
|
|
Generate [hash](https://en.wikipedia.org/wiki/Hash_function) over the last produced PoW, last produced proof and new proof, where proof is a prime number (to further increase [complexity](https://en.wikipedia.org/wiki/Computational_complexity) and [unpredictability](https://en.wikipedia.org/wiki/Predictability), this is a [random number](https://en.wikipedia.org/wiki/Random_number_generation) (between fixed MIN/MAX, however, the resulting prime can be larger than MAX) and the hash starts with n number of 0s (zeros).
|
|
|
|
|
|
#### Example (Genesis hash)
|
|
|
#### Example ([Genesis hash](https://en.bitcoin.it/wiki/Genesis_block))
|
|
|
|
|
|
``` python
|
|
|
last_block = {
|
... | ... | @@ -28,7 +30,7 @@ new_block = { |
|
|
|
|
|
#### Explanation of process
|
|
|
|
|
|
The hash generating process (aka publisher in MQ language), will send the generated "block" over to the message exchange. On the other side the validating processes (aka consumer in MQ language) will pop the messages from the queue and validate the "block". All blocks received on this end _must_ be valid, in order to pass this test.
|
|
|
The [hash](https://en.wikipedia.org/wiki/Cryptographic_hash_function) generating process (aka publisher in MQ language), will send the generated "block" over to the message exchange. On the other side the validating processes (aka consumer in MQ language) will dequeue one message after the other and validate each of these "block". All blocks received on this end _must_ be [valid](https://en.wikipedia.org/wiki/Validity), in order to pass this test.
|
|
|
|
|
|
|
|
|
## Scenario 2
|
... | ... | @@ -37,41 +39,77 @@ The hash generating process (aka publisher in MQ language), will send the genera |
|
|
|
|
|
### Producer
|
|
|
|
|
|
The generating process (aka publisher in MQ language), will just send a fixed hash to the message broker. The hash, however, contain, at least, UTF-8 characters, if not UTF-16. Since this is a fixed string, it can still be validated on the other side by the validating processes (aka consumer in MQ language).
|
|
|
The generating process (aka [publisher](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) in MQ language), will just send a fixed hash to the message broker. The hash, however, contain, at least, [UTF-8](https://en.wikipedia.org/wiki/UTF-8) characters, if not [UTF-16](https://en.wikipedia.org/wiki/UTF-16). Since this is a fixed string, it can still be validated on the other side by the validating processes (aka [consumer](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) in MQ language).
|
|
|
|
|
|
#### Example hash
|
|
|
``` python
|
|
|
'testdict': {
|
|
|
u'مفتاح': u' قيمة',
|
|
|
u'键': u'值',
|
|
|
u'キー': u'値'
|
|
|
u'مفتاح': u' قيمة', # Arabic; key: value
|
|
|
u'键': u'值', # Chinese; key: value
|
|
|
u'キー': u'値', # Japanese; key: value
|
|
|
}
|
|
|
```
|
|
|
|
|
|
#### Explanation of the test
|
|
|
|
|
|
Since the process sends non-ASCII characters over the channel, we can see if the encoding/decoding works correct or not. In theory this will also test if JSON module from Python does it's job correctly, but given our experience, this is usually not the problem.
|
|
|
Since the process sends non-[ASCII](https://en.wikipedia.org/wiki/ASCII) characters over the channel, we can see if the encoding/decoding works correct or not. In theory this will also test if [JSON](https://en.wikipedia.org/wiki/JSON) module from [Python](https://en.wikipedia.org/wiki/Python_(programming_language)) does it's job correctly, but given our experience, this is usually not the problem.
|
|
|
|
|
|
## Comparison
|
|
|
|
|
|
All these tests will be done with CloudAMQP as well.
|
|
|
All these tests will be done with [CloudAMQP](https://cloudamqp.com) as well (separately).
|
|
|
|
|
|
Note: Since we're using the free plan (aka ['Lemur'](https://customer.cloudamqp.com/instance/create?plan=lemur)) on [CloudAMQP](https://cloudamqp.com), your mileage may vary.
|
|
|
|
|
|
## Deployment of the test scripts
|
|
|
|
|
|
### Requirements
|
|
|
### Computing resources
|
|
|
|
|
|
Resource consumption shouldn't be too high. The scripts are not (multi-)[threaded](https://en.wikipedia.org/wiki/Thread_(computing) (See also: [Threaded code](https://en.wikipedia.org/wiki/Threaded_code) and will only consume at maximum one (logical) [core](https://en.wikipedia.org/wiki/Multi-core_processor) per (Python) [instance](https://en.wikipedia.org/wiki/Instance_(computer_science)). [Memory](https://en.wikipedia.org/wiki/Random-access_memory) and [disk space](https://en.wikipedia.org/wiki/Disk_storage) consumption can be neglected (only a few MB).
|
|
|
|
|
|
*TODO*
|
|
|
### Requirements
|
|
|
|
|
|
* Firewall open for accessing MQ server
|
|
|
* HTTP(s) Proxy (for pip)
|
|
|
* pika Python module
|
|
|
* [Firewall](https://en.wikipedia.org/wiki/Firewall_(computing)) open for accessing MQ server
|
|
|
- Depends a bit on the implementation/configuration, but usually AMQP uses [5672/tcp/udp](http://www.rabbitmq.com/uri-spec.html) and AMQP over TLS (aka AMQPs) uses [5671/tcp/udp](http://www.rabbitmq.com/uri-spec.html). See also [IANA](https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=AMQP) about the officially assigned port numbers.
|
|
|
* Python 2.x or 3.x
|
|
|
* Python [virtualenv](https://pypi.python.org/pypi/virtualenv)
|
|
|
* [HTTP(s) proxy](https://en.wikipedia.org/wiki/Proxy_server) (for [pip](https://pypi.python.org/pypi/pip))
|
|
|
* Pika Python AMQP Client Library (aka [pika](https://pypi.python.org/pypi/pika/))
|
|
|
|
|
|
### Steps to deploy
|
|
|
|
|
|
* Checkout source via git
|
|
|
* Checkout source via [git](https://git-scm.com/) (if git is not installed: ```yum install -y git``` (on RHEL))
|
|
|
* Create Python virtual environment
|
|
|
* Source virtualenv
|
|
|
* Install requirements (eg. using pip, since this is known to work)
|
|
|
* Change AMQP URL by exporting shell variable AMQP_URL
|
|
|
* Run as many publishers/consumer |
|
|
\ No newline at end of file |
|
|
* Source virtualenv (See also child section)
|
|
|
* Install requirements (eg. using pip, since this is known to work; See also child section)
|
|
|
* Change AMQP URL by exporting shell variable AMQP_URL or edit the code
|
|
|
* Run as many publishers/consumer as you like (eg. ```./producer.py``` or ```./validate.py```
|
|
|
|
|
|
#### Python virtualenv
|
|
|
|
|
|
Usually installed on [CentOS](https://www.centos.org/)/Red Hat Linux, aka [RHEL](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux) (6 and 7, untested on 5) via:
|
|
|
``` shell
|
|
|
yum install python-virtualenv
|
|
|
```
|
|
|
|
|
|
#### Python virtualenv
|
|
|
|
|
|
Usually this can be installed with:
|
|
|
``` shell
|
|
|
virtualenv <path/to/virtualenv>
|
|
|
```
|
|
|
|
|
|
##### Proxy
|
|
|
|
|
|
In case you need to use a http(s) proxy (for virtualenv/pip), export the following environment variable (bash):
|
|
|
``` bash
|
|
|
export http_proxy=http://<yourproxy.cmpny.tld>:<port>
|
|
|
export https_proxy=$http_proxy
|
|
|
virtualenv <path/to/virtualenv>
|
|
|
```
|
|
|
|
|
|
##### Python requirements
|
|
|
|
|
|
Python requirements are installed via:
|
|
|
``` shell
|
|
|
pip install -r requirements.txt
|
|
|
``` |
|
|
\ No newline at end of file |