The dramatic technological advances of recent decades offer opportunities to modernize the way in which water and other natural resources are planned and managed. New high-quality datasets covering topics like water resources, disasters, climate change, trade and general development are being collected and made public as online data and mapping services by institutions and governments around the world. Earth observation from an increasingly powerful fleet of public and private satellites provides unbiased, synoptic views of the globe. 
But many of these datasets – especially those from in-situ monitoring, even when digital – are fragmented, come in various formats, and are not well-known or easily accessible or viewable when scattered among thousands of websites. Communities, governments, regional organizations, development professionals, and others need easy, reliable, and useful access to these diverse datasets to support decisions at all levels, especially when it comes to managing watersheds. There is a need for a new paradigm to support decisions with open access data and tools that can leverage recent technological advances.
Need for a decision focus
Every day decisions are made – or not made (which also have impact) – relating to hydrometeorology. These can include decisions related to short-to-long term planning and daily operations:
How can my country be better prepared for adapting to historical climate variability and ongoing climate change? Who do we warn (and when) about the impending flood? Which stakeholders need to be compensated this year for drought losses? When do we release water from this dam to help cushion the impact of incoming floods? Will I have enough water to sustain my crop? How much groundwater to pump today for this field? How could land use and climate change impact the services of this watershed? Where could the water required for an urbanizing landscape come from? Do I need an umbrella today?
All these decisions can be taken in a manner that includes both meaningful analytical and stakeholder inputs – with the analysis providing the information and stakeholders providing the opinions.
The data value chain
Often, we get stuck at the level of data, seeking the decision nails that a shiny new data hammer can support. It is useful to change that paradigm by exploring the needs of the decision and how the data value chain can be developed to convert data to information to knowledge to support decisions.
For example, flood management decisions require good flood early warnings generated by forecasts (weather, hydrologic, and flood inundation) that require a range of data and analysis as shown.
Data, data, everywhere
There is an increasing amount of innovative equipment for monitoring and collecting data on all aspects of the hydrologic cycle – weather, surface and ground water hydrology, water quality, disaster risks, land cover, water use, and other water stocks, fluxes, and trends – in any watershed. The way these are transmitted to data repositories is also changing – moving from manual observations to automated digital systems. All of this is resulting in a deluge of petabytes of data, with just one Himawari meteorological satellite generating over 3 Terabytes every day.
Modelling tools help to convert these data into information – these can range from forecasting (e.g. short-term to seasonal), to water balance and systems simulation, optimization, and multi-criteria models that provide a more holistic systems perspective. Innovations in modelling tools are also making these more widely available.
What’s broken today?
However, many countries today rely on very distributed, “retail” level approaches to support administrative, watershed/basin decisions at a sub-national level. The data collected in this way (often still done manually on paper in less developed countries) has many challenges – primarily that they are not usually comprehensive in space and time, of uncertain quality, and not available or accessible in a timely manner in useful formats to support analysis and decision-making.
Analytical tools, if used, are often also at a “retail” level, coming at great cost for a small area of interest (e.g. a single watershed or river basin), based on expensive proprietary software that only a few donor-financed programs can afford, and are so complex only a few PhDs in the country can use them. Clients are often confused on which tools to use and the innovations are difficult to keep track of, leading to further confusion. Access to the tools and underlying data is often restricted by policies and cost/knowledge of the software requirements, making them usable by only a small team that has little incentive to change the status quo. It is difficult for clients in the developing world to effectively participate in developing, sustaining, and improving these tools or to integrate them into a decision-making framework that often has a legacy culture of “data-free analysis and analysis-free decision-making.”
The resulting institutional complexity has resulted in a fragmented approach that tends to encourage disintegrated water resources management – with high transaction costs for exchanging data even among government agencies. Multiple, fragmented observation networks blossom with poor access or efficiency of use of the data from these networks.
Emerging wholesale approaches
A range of more “wholesale” platforms that provide a glimpse into the future are emerging. These are often enabled by a few innovations:
- Data to online data services: The conversion of traditional databases to online data services is one of the most under-appreciated advances in water resources planning and management. The serving of data through open application programming interfaces and using harmonized standards for data and mapping services, especially based on Open Geospatial Consortium (OGC) formats, has enabled ingestion of such data for interactive, real-time visualization and online analysis. Using this approach, institutions can retain the right to serve and update the data that they are responsible for, while benefitting from other data services.
- Analysis to online analytical services: Similarly, a move to online analytical services is promising to revolutionize the toolkits for water managers and users even in the poorest of nations. These tools are not only using the power of online data services to ingest in-situ and earth observation products in real-time, but also providing their outputs using the same types of online data platforms.
- Get ahead in the Cloud: The emergence of high-speed cloud services has enabled a new paradigm for storing, analyzing, and accessing data. These tend to be a combination of commercial subscription services (e.g. cloud storage and cloud computing platforms) and experimental somewhat free storage/analytical platforms (e.g. Google Earth Engine), with a promise of artificial intelligence making these systems even smarter in the coming years. There is significant scope for development partners to work with information technology foundations to develop a free global cloud framework for hydromet and water data that could completely revolutionize water resources management in a compressed timeframe.
- Wholesale platforms: A particularly exciting innovation that has been developing is to move to more “wholesale” platforms that can help access data and analytical services across platforms at low cost. An example of such a platform from the World Bank Group is the Spatial Agent App (see page 22), that provides easy access to a growing world of free, public-domain data and analytical services on mobile devices.
An exciting future – with a range of increasingly powerful “wholesale” data, analytical, and knowledge services and platforms that can be applied at any “retail” level of decision support – lies ahead. Many of these will consist of free online services that can be packaged, augmented, and used in a new generation of customized portals, apps, and interactive e-books.
The level of connectivity in many parts of the world, especially Africa, does not permit easy access to this new world of data and analysis. However, the recent dramatic growth and projections for further growth in smartphone use and data connectivity worldwide and the amazing advances in disruptive digital technologies such as machine learning and other forms of artificial intelligence have the potential of bridging the digital and development divides. The leveraging of such advances will reduce transaction costs to get new products and services.
In addition to what will inevitably be a plethora of free data and tools for the digital earth, there will be a range of value-added data, analytical, alerts, and knowledge subscription or other services provided by the private sector and customized to various stakeholders. This will usher in a new era of cooperation across the “wholesale” and “retail” levels. Much operational work done at a local “retail” level can better leverage global and regional “wholesale” platforms and partnerships. “Retail” work can also benefit “wholesale” services by providing data for better calibration and validation and helping to shape better services and customization based on demand.
What do we need to do today other than wait for this new world? The answer lies in shaping mindsets to use the power of individuals and institutions to meaningfully “liberate” data in free, open, real-time, and analysis-ready service formats. Today, you would be hard-pressed to find a single hydrologic gauge that meets this criterion in many developing countries. And also in shaping mindsets that proactively try to learn from, leverage, and contribute to global good practice. Local leadership with open, curious minds are essential to undertake the difficult task of changing entrenched institutional inertia. These are the conditions that will enable a paradigm shift for resources management and climate resilience in watersheds around the world.
Spatial Agent App
A world of free, public domain data and analytical services
The Spatial Agent App from the World Bank (iOS, Android and web versions are available) puts a world of insights at the user’s fingertips by enabling easy and highly interactive access to a burgeoning group of free, public domain multi-sectoral datasets – including near real-time data – at the global, regional, and national levels.
The Spatial Agent platform brings together a wide range of development-related datasets from thousands of web services from several institutions around the world. Users can seamlessly search, visualize and compare data related to development issues through interactive maps and charts at different scales and time ranges around the world.
The system is being updated regularly with new data and analytical services, functionality, partnerships (including the use of data catalogs from the Group on Earth Observations), and user platforms. The aim is to improve awareness of free data services not only on hydromet aspects but also on other development aspects to give a more holistic multi-sectoral spatial perspective.
Such platforms could help showcase and leverage a rapidly exploding world of relevant free online data and analytical services on convenient mobile devices. They could also help institutions develop their own customized versions of Portals and Apps for their targeted clients.