Given multiple widespread stationary data sources such as ground-based sensors, an unmanned aircraft can fly over the sensors and gather the data via a wireless link. Performance criteria for such a network may incorporate costs such as trajectory length for the aircraft or the energy required by the sensors for radio transmission. Planning is hampered by the complex vehicle and communication dynamics and by uncertainty in the locations of sensors, so we develop a technique based on model-free learning. We present a stochastic optimisation method that allows the data-ferrying aircraft to optimise data collection trajectories through an unknown environment in situ, obviating the need for system identification. We compare two trajectory representations, one that learns near-optimal trajectories at low data requirements but that fails at high requirements, and one that gives up some performance in exchange for a data collection guarantee. With either encoding the ferry is able to learn significantly improved trajectories compared with alternative heuristics. To demonstrate the versatility of the model-free learning approach, we also learn a policy to minimise the radio transmission energy required by the sensor nodes, allowing prolonged network lifetime.
References
[1]
Baghzouz, M.; Devitt, D.A.; Fenstermaker, L.F.; Young, M.H. Monitoring vegetation phenological cycles in two different semi-arid environmental settings using a ground-based NDVI system: A potential approach to improve satellite data interpretation. Remote Sens 2010, 2, 990–1013, doi:10.3390/rs2040990.
[2]
McQuillen, H.L.; Brewer, L.W. Methodological considerations for monitoring wild bird nests using video technology. J. Field Ornithol 2000, 71, 167–172.
[3]
Muskett, R.R.; Romanovsky, V.E. Alaskan permafrost groundwater storage changes derived from GRACE and ground measurements. Remote Sens 2011, 3, 378–397, doi:10.3390/rs3020378.
[4]
Jenkins, A.; Henkel, D.; Brown, T. Sensor Data Collection through Gateways in a Highly Mobile Mesh Network. Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC), Hong Kong, China, 11– 15 March 2007; pp. 2784–2789.
[5]
Henkel, D.; Brown, T.X. Towards Autonomous Data Ferry Route Design through Reinforcement Learning. Proceedings of the 2008 International Symposium on a World of Wireless, Mobile and Multimedia Networks (WOWMOM), Newport Beach, CA, USA, 23–26 June 2008; pp. 1–6.
[6]
Gu, Y.; Bozdag, D.; Brewer, R.W.; Ekici, E. Data harvesting with mobile elements in wireless sensor networks. Comput. Netw 2006, 50, 3449–3465, doi:10.1016/j.comnet.2006.01.008.
[7]
Somasundara, A.A.; Ramamoorthy, A.; Srivastava, M.B. Mobile element scheduling with dynamic deadlines. IEEE Trans. Mobile Comput 2007, 6, 395–410, doi:10.1109/TMC.2007.57.
[8]
He, T.; won Lee, K.; Swami, A. Flying in the Dark: Controlling Autonomous Data Ferries with Partial Observations. Proceedings of the 11th ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc), Chicago, IL, USA, 20–24 September 2010; pp. 141–150.
[9]
Zhao, W.; Ammar, M.H. Message Ferrying: Proactive Routing in Highly-Partitioned Wireless Ad Hoc Networks. Proceedings of the The Ninth IEEE Workshop on Future Trends of Distributed Computing Systems (FTDCS ’03), San Juan, Puerto Rico, 28– 30 May 2003; pp. 308–314.
[10]
Dunbabin, M.; Corke, P.; Vasilescu, I.; Rus, D. Data Muling over Underwater Wireless Sensor Networks Using an Autonomous Underwater Vehicle. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Orlando, FL, USA, 15– 16 May 2006; pp. 2091–2098.
[11]
Bin Tariq, M.M.; Ammar, M.; Zegura, E. Message Ferry Route Design for Sparse Ad Hoc Networks with Mobile Nodes. Proceedings of the MobiHoc: 7th ACM International Symposium On Mobile Ad Hoc Networking and Computing, Florence, Italy, 22– 25 May 2006; pp. 37–48.
[12]
Ma, M.; Yang, Y. SenCar: An energy-efficient data gathering mechanism for large-scale multihop sensor networks. IEEE Trans. Parallel Distrib. Syst 2007, 18, 1476–1488, doi:10.1109/TPDS.2007.1070.
[13]
Tekdas, O.; Lim, J.; Terzis, A.; Isler, V. Using mobile robots to harvest data from sensor fields. IEEE Wirel. Commun. Special Issue Wirel. Commun. Netw. Robot 2008, 16, 22–28.
[14]
Sugihara, R.; Gupta, R.K. Improving the Data Delivery Latency in Sensor Networks with Controlled Mobility. Proceedings of the 4th IEEE International Conference on Distributed Computing in Sensor Systems (DCOSS), Santorini Island, Greece, 11– 14 June 2008; pp. 386–399.
[15]
Sugihara, R.; Gupta, R.K. Path Planning of Data Mules in Sensor Networks; ACM: New York, NY, USA, 2011; Volume 8, pp. 1–27.
[16]
Henkel, D.; Brown, T.X. On Controlled Node Mobility in Delay-Tolerant Networks of Unmanned Aerial Vehicles. Proceedings of the International Symposium on Advanced Radio Technolgoies, Boulder, CO, USA, 2–4 June 2008; pp. 7–16.
[17]
Carfang, A.; Frew, E.W.; Brown, T.X. Improved Delay-Tolerant Communication by Considering Radio Propagation in Planning Data Ferry Navigation. Proceedings of the AIAA Guidance, Navigation, and Control, Toronto, ON, Canada, 2–5 August 2010; pp. 5322–5335.
[18]
Stachura, M.; Carfang, A.; Frew, E.W. Cooperative Target Tracking with a Communication Limited Active Sensor Network. Proceedings of the International Workshop on Robotic Wireless Sensor Networks, Marina Del Rey, CA, USA, 8–10 June 2009.
[19]
Jiang, F.; Swindlehurst, A.L. Optimization of UAV heading for the ground-to-air uplink. IEEE J. Sel. Areas Commun 2012, 30, 993–1005, doi:10.1109/JSAC.2012.120614.
[20]
Jun, H.; Zhao, W.; Ammar, M.H.; Zegura, E.W.; Lee, C. Trading latency for energy in densely deployed wireless ad hoc networks using message ferrying. Ad Hoc Netw 2007, 5, 444–461, doi:10.1016/j.adhoc.2006.02.001.
[21]
Anastasi, G.; Conti, M.; Di Francesco, M. Reliable and energy-efficient data collection in sparse sensor networks with mobile elements. Perform. Eval 2009, 66, 791–810, doi:10.1016/j.peva.2009.08.005.
[22]
Sugihara, R.; Gupta, R.K. Optimizing Energy-Latency Trade-off in Sensor Networks with Controlled Mobility. Proceedings of the IEEE INFOCOM Mini-Conference, Rio de Janeiro, Brazil, 19– 25 April 2009; pp. 2566–2570.
[23]
Ciullo, D.; Celik, G.; Modiano, E. Minimizing Transmission Energy in Sensor Networks via Trajectory Control. Proceedings of the IEEE Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), Avignon, France, 31 May– 4 June 2010; pp. 132–141.
[24]
B？l？ni, L.; Turgut, D. Should I send now or send later? A decision-theoretic approach to transmission scheduling in sensor networks with mobile sinks. Wirel. Commun. Mobile Comput 2008, 8, 385–403, doi:10.1002/wcm.584.
Anastasi, G.; Conti, M.; Di Francesco, M.; Passarella, A. Energy conservation in wireless sensor networks: A survey. Ad Hoc Netw 2009, 7, 537–568, doi:10.1016/j.adhoc.2008.06.003.
[27]
Watts, A.C.; Ambrosia, V.G.; Hinkley, E.A. Unmanned aircraft systems in remote sensing and scientific research: Classification and considerations of use. Remote Sens 2012, 4, 1671–1692, doi:10.3390/rs4061671.
[28]
Pearre, B.; Brown, T.X. Fast, Scalable, Model-free Trajectory Optimization for Wireless Data Ferries. Proceedings of the IEEE International Conference on Computer Communications and Networks (ICCCN), Maui, HI, USA, 31 July– 4 August 2011; pp. 370–377.
[29]
Wagle, N.; Frew, E.W. A Particle Filter Approach to WiFi Target Localization. Proceedings of the AIAA Guidance, Navigation, and Control Conference, Toronto, ON, Canada, 2–5 August 2010; pp. 2287–2298.
[30]
Sadegh, P.; Spall, J. Optimal Random Perturbations for Stochastic Approximation Using a Simultaneous Perturbation Gradient Approximation. Proceedings of the American Control Conference, Albuquerque, NM, USA, 4–6 June 1997; pp. 3582–3586.
[31]
Hirokami, T.; Maeda, Y.; Tsukada, H. Parameter estimation using simultaneous perturbation stochastic approximation. Electr. Eng. Jpn 2006, 154, 30–39, doi:10.1002/eej.20239.
[32]
Kohl, N.; Stone, P. Policy Gradient Reinforcement Learning for Fast Quadrupedal Locomotion. Proceedings of the IEEE International Conference on Robotics and Automation, New Orleans, LA, USA, 26 April– 1 May 2004; pp. 2619–2624.
[33]
Peters, J.; Schaal, S. Reinforcement learning of motor skills with policy gradients. Neural Netw 2008, 21, 682–697, doi:10.1016/j.neunet.2008.02.003. 18482830
[34]
Roberts, J.W.; Moret, L.; Zhang, J.; Tedrake, R. Motor Learning at Intermediate Reynolds Number: Experiments with Policy Gradient on the Flapping Flight of a Rigid Wing. In From Motor to Interaction Learning in Robots; Sigaud, O., Peters, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2009.
[35]
Pearre, B.; Brown, T. Model-free Trajectory Optimization for Wireless Data Ferries among Multiple Sources. Proceedings of the Globecom Workshop on Wireless Networking for Unmanned Aerial Vehicles (Wi-UAV), Miami, FL, USA, 6–10 December 2010.
[36]
Glynn, P. Likelihood Ratio Gradient Estimation: An Overview. Proceedings of the 1987 Winter Simulation Conference, Atlanta, GA, USA, 14–16 December 1987; pp. 366–375.
Sutton, R.S.; McAllester, D.; Singh, S.; Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Process. Syst 2000, 12, 1057–1063.
[39]
Baxter, J.; Bartlett, P.L. Infinite-horizon policy-gradient estimation. J. Artif. Intell. Res 2001, 15, 319–350, doi:10.1016/S0954-1810(01)00028-0.
[40]
Tang, J.; Abbeel, P. On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient. Proceedings of Twenty-Fourth Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 6–11 December 2010.
