How we superseded Army's expectations with a Drone Op

This is a two part blog post. Part 1 talks about the background to the entire exercise, the Part 2 talks about the simulation exercise which was built on top of the entire Drone Op generated data.
 
The background to this entire operation were our simulation activities that we were doing for Defence. Our past experience had mostly been with the Indian Air Force, and recently had also done some aviation simulation works for the Indian Navy. However when we demonstrated the same to the Army, the feedback given was ‘Well, all this simulation, terrain recreation etc is fine for aircrafts flying thousands of feet above the ground, but we need something which can give a soldier a feeling of if he has the boots on the ground’. This needed a very highly accurate ground mapping which required two elements – a digital elevation map (known as DEM) and ground texture.
 
Now DEMs were available, but those in the public domain had extremely low resolutions, only suited for very high-altitude passes, like those available from NASA and Japan Space Systems. Higher resolution DEM imagery is either available only to military (including ISRO’s own DEM data – but we didn’t have access to that), or very expensive to buy for commercial use.
 
Google earth (including Pro) pack – while allowed you to export imagery, it didn’t allow you to export the elevation, and even imagery had to be stamped with Google earth’s logo. And the resolution wasn’t good that we wanted anyway.  
 
The target that had been give to us for the textures resolution was – 10 cm per pixel. A massively ambitious target. But audacious goals only embolden us to achieve them. We decided, we’d get our DEM via an aerial survey.
 

Choice of platform

In order to do an aerial DEM, we obviously needed an unmanned aerial system (UAS), or commonly known as – Drone. There were many choices and factors

1. Fixed wing (like your typical aircraft configuration) or Rotory
2. Within rotory – tricopter/quadcopter/hexacopter/octacopter
3. Choice of frame – plastic, carbon fibre etc.
4. Choice of motors and propellers, and electronic speed drives to run them.
5. Choice of controller – This was a major choice, with top two contenders being DJI vs APM (Ardupilot). This is like a choice between Apple (proprietary, closed system) vs Android (open source system)
6. Payload – which cameras to take, including whether to use gimbals for stabilization or not.
7. Final processing – after the data has been captured, how to process it and convert it into useful format ? 

Learning to fly

The first drone we had, we got a pre-assembled one based on the APM platform. The supplier had assured us that it was in ready to fly configuration, and you only had to power it up, and it was fit to fly. It wasn’t so easy as it turned out.
 

While we were very careful in powering it up, and had gone through all the relevant manuals, including some drone Sims, the drone just kept going rogue. We were unable to fly it either manually, or using the automatic mode. And with every crash, we had part breakdowns – most notably of propellers, and some cuts and bruises from prop hits.

We got to know the local drone parts supply ecosystems pretty well, which was vital to our later exercises. What we couldn’t get – we made. Our 3D printer was an invaluable tool, making prop-guards or coupling mechanisms so that we could fit those payloads which weren’t otherwise fitting on our Drone’s chasis.
 
But we still needed to get better at flying a drone, and for this, we got a miniscule quadcopter and the rotory flying guide – and learnt to fly this very gently – starting with a hop up, and then touch down, to eventually making it go from Pt. A to Pt. B as we wanted it to, mustering up our confidence in handling the actual operation.

Choice of location

Just about when we were considering the above and buying the relevant parts – an important decision was where to actually execute the exercise. We were located in New Delhi, but flying a drone within Delhi was a serious no go – there had been reports of sighting rogue drones in news, including at the airport, and in such a charged atmosphere, we didn’t want to get a system airborne in and around Delhi. So we decided to head out, and choose a location which

1. More matched the typical terrain where army may actually have to maneuver, including with troops and armor.
2. A location which had far less security sensitivity, or where we at least knew the locals including the administration.
3. A location which could be positively identified both on ground, in maps, and in Goggle earth.

And we knew one such exact location – the meteorological laboratory in the middle of agricultural fields typical of the northwestern plains, in the city of Hisar – which also had an army cantonment with tanks units plenty. 
 
So we loaded our car with equipment, drone supplies, electronics elements, our PCs, and off we were, to see what we could come up with. ​

Initial test runs

​We started doing test runs, in order to test the upper ceiling, battery endurance, signal strength etc. We’d typically carry two charges full of battery, and aim to do 2 flights per morning and evening, spending the intermediate time learning, tweaking, and charging ourselves and the batteries.  Joshua would be at the controls, while Karan would monitor the telemetry (data) on an android tablet.

Meanwhile we scouted the area and identified a 50 acre area, which is about 400m by ½ km area that we’d be surveying.  We were typically getting an endurance of 15 minute on a full battery charge, and we needed to decide how to comb through the entire area. We needed to make full length passes to cover the entire areas, but the pass to pass ‘offset’ was a crucial setting. If we keep the offset too much, we’d get lower number of passes and cover the area within stipulated flying time, but may not get enough image overlap between each image to facilitate stitching to get a good terrain. On the other hand, if we kept the offset small enough to get enough overlap between images, then the number of passes increased, and we wouldn’t have enough endurance to last the flight.

Now that we knew our systems parameters well, and with help from basic trigonometry, we were able to arrive on an optimized solution which ensured that we could cover the 50+ acres within our endurance flying time of 15 minutes. So the next morning was decided to be D-Day, and batteries were left to charge. ​

DDay and Crash

Having programmed the path along the GPS coordinates, with final testing of the UAV, gimbal and camera, batteries etc, all systems were go at the next morning and the mission was put into practice. The UAV took off beautifully, rose to the programmed height, oriented itself and started the scan. We were all positioned at the extreme corners of the 400m x 500m observing the UAV carry out its mission. It all seemed to be going as per plan.  However, the winds started picking up. The UAV had to move across the ground at about 20km/hour to cover the required area in the stipulated time (and the camera was also programmed to take pictures at a fixed frequency) so that image-to-image distance was within the ‘overlap-parameters’.  

But with risings winds, the UAV had to work extra. It got the benefit of tailwinds along one pass, but faced headwinds along the opposite pass – and the results didn’t cancel each other out – overall, the effect is reduction of endurance.
 
And on the last pass, either the wind became too much to handle, or the battery ran out (it should have returned to home, in case battery was low, but it didn’t), the UAV decided it had had enough, made a final gesture of salute by gaining a bit of altitude, and then stopped mid air before plummeting to mother earth – choosing the only spot in the entire 50 acre area which had a bricked flooring.

We rushed to pickup the pieces – the props were gone, the frame was broken. The gimbal twisted its precision metal brackets. Almost all motors on both the UAV and the gimbal were fried. Battery had ballooned. Luckily, the camera and the APM module survived – so did the data captured by them.

Drone repair works

We’d carried enough spares with us including alternate platforms to get back on the job. So we took the parts which were functioning, and assembled them on a new frame to have a new working APM drone, and also parallelly assembled a DJI one. Our house turned into a drone repair and maintenance depot, with all types of mechanical and electronic spares related to drone lying around. 

Few words on DJI

DJI’s phantom drone is the easiest available commercial drone – and you can see the same being flown even in weddings. When we were shopping for parts – we happened to open a DJI and look inside – and it seemed like a bad job of engineering, including cables running without order and poorly assembled. We instead decided to keep our own frame, motors etc, and go for DJI control boards – known as the Naza. However, the Naza wouldn’t fly by GPS coordinates (only manually), so we had to invest in some more upgrades – notably the Bluetooth control module - one end connected at the drone and the other connected with either the PC or an iOS device, with which you could control it via a DJI app.
 
While by now we had enough experience building/breaking/maintaining a drone – the DJI proved itself quite challenging when it came for the telemetry to work. The radio modules wouldn’t just pair with each other – they had some very complex sequence of starting up which wasn’t explained very well in manuals, and after searching through numerous online forums, and by one all-nighter from Joshua – did we end up getting the remote telephony and telemetry work – but after that, we could fly the DJI powered drone with our iPad.

Meanwhile, the APM was also refurbished, and was up and running – so it was back to the meteorological center, where once again, the drones took to the air. ​

Post Processing

We are able to perceive depth because of our two eyes (called stereo vision). Similarly, DEM data is generated by satellites by having sensors looking at earth separated by a long boom, and the parallax between the two sensors is what gives the perception of depth. In our case, while we didn’t have stereo cameras – we did have image of the ground from multiple angles, and for each shot (after syncing the clock of camera and the drone), for each image we also had its exact position and orientation in space. Between the three drones, and multiple passes at the target area – by now we had enough imagery, and drone fly pass data, that we could now put it into post-processing.

​Now to do this entire data-processing, the options including commercial software as well as open-source. While open-source softwares  were a feasibility, we decided to test a proprietary software called Pix4D, which offered a trial version of two licenses. One license was used on a laptop at the field to quick test the data, and the other license was used at our amazingly powerful PC called Janus – which had a server level motherboard, with two Xeon processors each with 6 cores, and a 512 core GPU (Pix4D can harness parallel processing power of CUDA compatible GPUS).  Such massive computation power is needed for complex vision processing requirements like stitching such large number of images, and generating point cloud data in 3D.

Results

We were given a target of scanning about a 50 Acre area, with an image resolution of 10cm / pixel. We superseded that expectation by a factor of 2, and had a resolution of 5cm / pixel. And we had digital elevation map including a point cloud data, which we had a resolution of less than a feet (about 250mm), compared to the open source 90M resolution data which is normally even used in Google earth. That is a 360 times the resolution in height map data !  

Here are the results of the ortho-mosaic (stitched image data) overlaid on top of Google Maps and compared. You can see vivid details of the buildings, the marks of the furrows, and even the surf in the freshly irrigated field.  

And here is the point-cloud data - which is 3D reconstruction of the scene. You can easily establish the features of the terrain - the depth of the ditch, the height of the trees - even the measurement of the car parked there. 

On the whole – it was mission successful ! Thanks to the enduring efforts of the duo of Joshua and Karan. ​

Learning

While the entire operation was a huge learning exercise, here are a few things which we’d do differently the next time:

1. We’d use a flying wing – the huge surface area gives good lift, so the motors and propeller are working to keep the UAV flying through air than working against gravity. Thus, we can get a far higher endurance for the same battery.
2. No need to use a gimbal – since the flying wing mostly flies parallel to the ground (without tilting the was a quadcopter does), we don’t need secondary stabilization on the camera. In any case, the flight data has the ‘orientation’ of the camera, something which is taken care in post-processing.
3. Very importantly – we’d use Pixhawk. This is advanced version of the APM and is much more stable and fool-proof, and has more features (one of which will be an airspeed sensor).
4. We’d use a camera of even higher resolution – thus allowing us to go higher, cover more area, reduce our needed offset between the combing runs, and further increase the area that can be covered in a flying run.
5. And we’d time our run and finish it before the winds pick up (ideally, as early in the morning as possible)
6. Can even consider going for a drone that is officially endorsed by Pix4D – to get a joint proven package (but is costly). 

DGCA Drone operators guidelines.

​Sometime after our UAV operation for the Indian Army, the Directorate General of Civil Aviation - DGCA came out with draft guidelines governing drone flying in India. The essential part is, beyond a certain height - that is 200feet above ground level , flight clearance to be taken like for a real aircraft flying, and all drone pilots need to be registered pilots with DGCA.

​Interestingly, we got the last bit covered, as we have two licensed pilots in our organisation.  

Part 2 of this blog post, covers the simulation activity and training which happened on top of the generated terrain data.