Geoprocessing | Hugo Roussaffa a remote geomatic expert

Indexing all environmental information on an H3 vector grid

Sat, 01 Jan 2022 00:00:00 +0000

Project description

The goal is to build a decision support system based on geographic and environmental data: wildfires, land cover, administrative boundaries, drought indicators, light pollution, and more. The data is indexed on an H3 grid for fine-grained spatial analysis. Processing is carried out with Python algorithms and the datamarts are generated with DBT, a SQL generator based on Jinja templates.

Our project stands out from standard business intelligence projects through its ability to integrate geographic and environmental data, to perform fine-grained spatial analysis thanks to H3 indexing, and to stay flexible enough to meet specific needs.

H3 is a hexagonal spatial indexing system developed by Uber that divides the Earth’s surface into a grid of hexagons of variable sizes. It is particularly useful for fine-grained spatial analysis of data, as it allows data to be stored at different levels of granularity.

The advantages of using H3 over a standard geographic information system are numerous:

It adapts the fineness of the spatial analysis, since data can be stored at different levels of spatial granularity.
It is also more efficient in terms of data storage, as it can store data of different sizes in a single grid.
It works easily with visualization tools such as geographic dashboards, giving users a better understanding of the data.

Using H3, we can better understand the risks related to wildfires, drought and light pollution. H3 indexing enables fine-grained spatial analysis, meaning we can identify the areas most affected by drought or light pollution with greater accuracy.

DBT is a key tool for our project because it lets us generate datamarts from our data. It uses Jinja templates to generate SQL queries that can be used to build geographic dashboards and automated reports. This gives users a better understanding of the data and enables more effective decision-making.

In short, our project is based on geographic and environmental data, fine-grained spatial analysis through H3 indexing, and flexibility to meet specific needs, using DBT to generate datamarts, geographic dashboards and automated reports.

Skills used and deliverables

Python
Data Visualisation
Data Management

Large-scale environmental geoprocessing and spatial analysis

Sat, 01 Jan 2022 00:00:00 +0000

What does Bilbo mean?

Bilbo stands for Business Intelligence for Land and Biodiversity Observation. It is a research and development project aimed at developing indicators for Earth observation.

Project objective

The environmental observatory’s goal is to translate the state, pressures and responses affecting the environment in New Caledonia. To meet these objectives, we collect and analyze environmental monitoring data produced locally by local authorities, mining companies, research institutes, associations and businesses, as well as data produced worldwide, such as that obtained from Earth observation satellites:

land cover and erosion,
wildfires,
vegetation drought indicators,
the light pollution indicator
…

The amount of information to be analyzed is massive and constantly growing — this is Big Data. To optimize our productivity in monitoring, we aim to automate processing and the publication of information in a decision-ready format.

Main challenges

Heavy, time-consuming geoprocessing

Spatial relationships

In a classic (non-spatial) BI project, the added value comes from crossing data in a model structured by relationships between information: reference keys. In our case, the relationship between information is ALSO spatial, and this specificity implies heavier calculations than key-based relationships.

From GIS to a decision support system

The approach we have followed so far consists of transposing the model of a Geographic Information System into a Decision Support System, which means defining which facts and which dimensions will allow us to answer all our questions.

The facts

Our first results led to the production of data made up of the results of geographic intersections between environmental pressures (burned areas, for example), context information (land cover types) and administrative perimeter information (municipal boundaries). The result of these divisions constitutes our DataWareHouse, which provides aggregated results according to our temporal, thematic and above all spatial dimensions.

This model is not fully satisfactory because it multiplies the crossings performed for each of the themes analyzed. Databases are weighed down by redundant information, notably due to the inclusive nature of the data depending on the scales studied (a region encompasses +/- several municipalities). Moreover, the heterogeneity of the precision of the data we cross sometimes leads to a patchwork of objects producing geometric invalidities or semantic inconsistencies.

The DataMarts (DTM)

These are the tables resulting from the aggregation of facts according to the chosen dimensions. These tables are then used to produce reports, maps and statistical analyses. They are produced by SQL scripts and stored in a relational database. The SQL scripts are redundant, so the use of the DBT framework brings an improvement in terms of maintainability and readability of the scripts.

Towards a more efficient approach

The side effects of our current approach led me to think about a more efficient, more robust and more flexible approach: the use of a lightweight spatial index that optimizes spatial relationship calculations. The following article presents this approach:

Indexing all environmental information on an H3 vector grid

Vegetation drought monitoring by remote sensing

Sat, 01 Jan 2022 00:00:00 +0000

This project follows on from the work carried out under the Space Climate Observatory project, named EO4DroughtMonitoring1 in New Caledonia, with the main objective of demonstrating the applicability and usefulness of such a tool in other Pacific island territories. The overall approach is to have a process adapted to all Pacific island territories that could subsequently be delivered as an open-access operational service in partnership with the region’s key stakeholders.

In the context of this project, we carried out:

the construction of a vegetation drought indicator capable of characterizing the current situation as well as historical and forecast situations, depending on the data available from the various satellite platforms and meteorological instruments of the territories concerned,
the production deployment of the indicator in New Caledonia, with the aim of demonstrating the operational nature of the solution,
the adaptation of the production chain for certain territories in the Pacific region,
the promotion of the data through interfaces and consultation and distribution services.

View the full interactive report by clicking this link