This project involves object recognition with Darknet and OpenCV's YOLO algorithm running on a distributed system for object tracking.
Server-side:
- @pjreddie Darknet YOLO (Already included in this repo) - https://github.com/pjreddie/darknet/
- .cfg that are in this repository have been tweaked to be faster with less powerful NVIDIA GPUs
- yolov3.weights + yolov3-tiny.weights files are required and need to be placed inside Server/darknet/
Client-side:
-
OpenCV 3.4.5 - https://github.com/opencv/opencv/releases/tag/3.4.5
- OpenCV_DNN additional modules - https://github.com/opencv/opencv_contrib/releases/tag/3.4.5
- Installation guide here: https://docs.opencv.org/3.2.0/de/d25/tutorial_dnn_build.html
- If you feel brave enough, you can try the automatic installation launching the script that you find inside this repo (it should download, compile the right version of OpenCV):
NB: This script doesn't install OpenCV. After executing it, you still have to go to opencv-3.4.5/build/ and
sudo sh build_opencv.sh
install.
-
A LOT of patience
After cloning this repository (git clone https://github.com/DPons97/reimagined-octo-spork.git ):
-
Build darknet inside
Server/darknet/(and test it!)
Download and place darknet's weights in Server/darknet/:wget https://pjreddie.com/media/files/yolov3.weights wget https://pjreddie.com/media/files/yolov3-tiny.weights
-
Build OctoSpork:
cd [...]/reimagined-octo-spork/ mkdir build cmake .. make [leave empty if you want to build everything. Otherwise, see below]
If you want to only build Server, change
makeline to:make OctoSporkServer
Otherwise, to build Client only:
make OctoSporkClient
-
If you decided to build Client, you must move and compile bkgSubtraction and nodeTracker inside Client/Executables.
cd into the build directory in reimagined-octo-spork
cd path-to/reimagined-octo-spork/build-
Server side:
./OctoSporkServer
-
Client side: There are two ways to run the client executable:
- Passing all information as arguments
./OctoSporkClient [ipaddress] [port] [node_id] [node_x] [node_y] [theta] [top_neighbour] [bottom_n] [left_n] [right_n]
- Passing a configuration file
./OctoSporkClient [path_to_cfg_file]
The configuration file should be a one-line file with the arguments you would pass as above.
NB: node_x, node_y, theta are respectively the coordinates and the phase displacement of the camera's node.
Run ROS demo client:
- Node 1 (set cam1 in file path in Client/Executables/ files)
./OctoSporkClient [ipaddress] 51297 1 0 0 0 3 4 5 2
- Node 2 (set cam2 in file path in Client/Executables/ files)
./OctoSporkClient [ipaddress] 51297 2 0 0 260 7 8 1 9
- Odroid xu4
- Jetson-TX2 with Nvidia Tegra technology
- Any PC with any Linux distribution (Tested on Fedora and Ubuntu)
The task for this project was to work on a framework for distributed software. Requirements included:
- Achieving a good level of customizability and scalability
- Compatible with mobile CPUs (ARM 32bit) and embedded boards
- Working demo of a case study using the framework
The initial goal was to develop a peer to peer system with an external resource controller that would manage the load between the nodes.
We took an incremental approach to the project and we first worked to build a simpler version with a client server architecture. We then realized that we could get a good result by improving our first version, and since time was running low, so we did.
As we needed to work on a realistic case study, we thought about diving into image recognition and building a distributed tracking system.
First, we did some research to find the algorithms we had at disposal and to select which one was the most suitable for our hardware.
Pjreddie's darknet network is really good concerning detection and it's fast enough on the Jetson but, having to also use an Odroid XU4 (8 cores CPU, no GPU), YOLO's library would result in really high computational times.
Here's where OpenCV's Deep Neural Network module comes in handy: it features a 9x faster implementation of DNN using CPU with the same darknet's yolov3 configuration file.
Some tweaking to the cfg are required to achieve an acceptable speed, although trading off some accuracy.
Now let's talk about the fun stuff.
The ROS system's architecture is based on one SERVER (in our case the Jetson-TX2) and one or more CLIENTS / NODES (Odroid XU4 or any linux pc).
Every client connects to the server sending his planimetry information (his node's ID and his neighbours' IDs) and waits for new instructions to be executed.
Client nodes relay on a text file to match the instruction ID received from the server with an actual executable that it can run in a new process. In this way the client side is easy to customize (more on that later...).
Every time a client connects to the server, the latter updates the planimetry of the system and communicates the node an "idle" operation that is, in our case study, a Background subtraction process.
When a client ends its idle operation, it communicates the termination (and a result, if present) to the server. The server then decides which task to assign to the client.
The server also stores all the PID of the processes running on clients. In this way, when necessary, it can tell a client to stop a certain process.
NB: If you are not into detection and tracking systems, there still is something for you. Just skip the next paragraphs and go to Customizing ROS.
Icons made by: Freepik, photo3idea-studio, eucalyp from www.flaticon.com
NB: As a lot of people don't have multiple cameras at their disposal, we implemented the video stream as a series of images (https://trac.ffmpeg.org/wiki/Create%20a%20thumbnail%20image%20every%20X%20seconds%20of%20the%20video).
Just remember to change FPS and other parameters inside bkgSubtraction and nodeTracker.
During this initial phase every client applies background subtraction (provided by OpenCV) to a given video stream.
Once a blob that is big enough is detected, the node sends last frame that was analysed to the server to run a first object detection in search of certain user-defined objects.
If something is found (inside our project we search for people), a new Tracking instruction is sent to the node that found the blob.
If a client receives this instruction, it starts tracking the defined object that should be in his sight.
For every frame that has the object in it, the node saves detected box's coordinates and tries to estimate distance from the camera (with bad results, for now 😐).
When the object is no more in sight of the client's camera, the server receives all saved coordinates and analyse them to decide whether it could keep tracking through other connected cameras.
This is possible thanks to the planimetry that is stored inside server.
Customization was our focus through the development of the project.
The full documentation of our code can be found here
As mentioned above, nodes customization is achieved through a map file that associates an ID with a corresponding executable (Client/Executables/executables.txt).
When the client receives a message it first checks if it's asking to kill a process or to start a new one.
In case it needs to start a new one, it looks up the corresponding executable and parses the parameters (if present).
It then opens a new socket connection to bind to the new process. At this point the client forks and the child will start the executable.
Eventually the main process communicates the pid of the new task to the server and starts to listen for new instructions again.
The ClientNode should not need any modifications, it just parses messages and set up the new task.
How to structure new executables:
New executables always take at least one parameters, the socket with which it communicates with the server.
If you want you can add additional parameters. The server will communicate them to the task when sending the execution instruction to the client.
The client could receive a message from the server asking to kill a given pid. In this case ClientNode will send a SIGTERM to the received pid (if it still running). You may want your executables to handle this signal in order to perform a clean exit of the task.
Unlike client, ROS Server has to be customized directly from the source code, as it's composed of only one generic class: Instruction.cpp (you can find source code inside Server/Instructions).
Basically, an object of type Instruction is provided with all basic functionalities to communicate with a ClientNode:
-
Default initializer:
Instruction(const string &name, std::map<int, int> &instructions, vector<void*> sharedMemory);
name is, as the name says, a symbolic identifier of the instruction.
instructions is a map that contains tuples <pid, socket>. Map key is the client process PID that's running the instruction bound to a specific socket (map value socket).
To allow communication between different nodes and instructions inside the same server, a generic optional sharedMemory can be passed as parameter (e.g. planimetry). Remember to cast this to the right data type before using!
-
First function to call every time you want to start the instruction is:
virtual void start(int socket, int port, std::vector<std::string> args);
This will let the instruction know which socket and port are bound to this node.
Additional arguments (args) can be passed if needed (optional).
-
To send a start signal for a new instruction inside a specific connected node:
int startInstruction(int instrCode, std::vector<string> args = std::vector<string>());
Where instrCode is the instruction ID you defined inside Client/Executables bound to the relative executable, and args are all arguments you want the node to receive (e.g. inside tracking, the object ID to track is one additional argument passed).
The return value is a new socket, which binds server to current node executable.
-
If during the execution of your program you need to receive an image, you can use:
bool getAnswerImg(int rcvSocket, cv::Mat& outMat);
Here rcvSocket is the receving socket (90% of the times it'll be your instruction socket), whereas outMat is a reference to a new OpenCV Mat where the received image will be stored.
-
To stop (or force) a specific instruction and disconnect the relative socket:
void disconnect(int instrPid = 0);
Where instrPid is the instruction node PID (if no pid is defined, all instructions will be closed of this node).
For additional informations and a code example, feel free to explore this repository.
Otherwise, if you have questions or something wasn't clear, you can contact us on GitHub or via email.
- Luca Collini @Lucaz97 - luca.collini@mail.polimi.it
- Davide Pons @DPons97 - davide.pons@mail.polimi.it