OVERTURE BETA TESTING FEEDBACK

[seanmcilroy29]

👋 Overture beta testing feedback

With our move to beta, we're asking users for some specific feedback for each of the Overture data themes.

Base:

Overture Base data consists of global coverage from OSM for land and water features that aren’t included in other Overture themes, but are necessary to render a complete basemap. For your use cases, is there anything critical missing from the Base theme that also isn’t captured in other Overture themes?

Buildings:

We’d like to hear your feedback on how the integration of multiple sources (OSM, Esri, Microsoft, Google) impacts your use case. For example, some regions may have a mix of all four sources of data. Does this introduce any useability challenges in your workflow, and in particular does mixing OSM and ML-derived building data result in any significant quality issues?

Places:

Tell us about your experience using Overture Places data for local search functions. For example, find a place by name (e.g. “La Vita e Bella Pizzeria”) or find places in a particular category (e.g. restaurants near me). We’re also interested in data issues you discover during testing including invalid/non-existent Places, Places that are out of business, and Places that are incorrectly positioned. We are particularly interested in records with high confidence scores that appear to be incorrect.

Transportation:

Tell us about your experience using Overture Transportation data for routing. More specifically, we’re interested in feedback about how you’re using the linear referencing system, turn restrictions, and access restrictions.

NOTE: If you’d like to share your feedback for any of these cases, please submit a new discussion in the appropriate category (e.g. Buildings, Transportation) and tag it with the “beta testing” label.

[acgeospatial]

Hello,

When I run the query from the documentation https://docs.overturemaps.org/getting-data/cloud-services/#13/47.6/-122.33/0/45

SELECT id,
class,
CAST(sources AS JSON) AS sources,
sources[1].dataset AS primary_source,
CAST(names AS JSON) AS names,
ST_GeomFromBinary(geometry) AS geometry
FROM
overture.release.v2024_04_16_beta_0
WHERE theme='places'
AND type='place'
AND bbox.xmin > -122.44
AND bbox.xmax < -122.25
AND bbox.ymin > 47.56
AND bbox.ymax < 47.71

on Athena. I appear to get 2 results back. On the previous version Alpha I was getting 26,023. What have I done wrong?

[acgeospatial]

SELECT id,
class,
CAST(sources AS JSON) AS sources,
sources[1].dataset AS primary_source,
CAST(names AS JSON) AS names,
ST_GeomFromBinary(geometry) AS geometry
FROM
overture.release.v2024_03_12_alpha_0
WHERE theme='places'
AND type='place'
AND bbox.minX > -122.44
AND bbox.maxX < -122.25
AND bbox.minY > 47.56
AND bbox.maxX < 47.71

On the last alpha actually returns 47,023 results. I must be doing something wrong. Can you advise?

[jwass]

@acgeospatial Thanks for writing this up. This has me stumped and I'm at the point of concluding this is an Athena bug - which would be very surprising. I looked at this example that pulls out a place that's in the dataset and is missing from your query results: id 08f28d55428a025a03b6d066707793b4.

SELECT id,
	st_astext(st_geomfrombinary(geometry)) AS geom,
	bbox,
	(
		bbox.xmin > -122.44
		AND bbox.xmax < -122.25
		AND bbox.ymin > 47.56
		AND bbox.ymax < 47.71
	) AS in_bbox
FROM overture.release.v2024_04_16_beta_0
WHERE theme = 'places'
	AND type = 'place'
	AND id = '08f28d55428a025a03b6d066707793b4'
-- 	AND bbox.xmin > -122.44
-- 	AND bbox.xmax < -122.25
-- 	AND bbox.ymin > 47.56
-- 	AND bbox.ymax < 47.71

returns one row as I'd expect

#	id	geom	bbox	in_bbox
1	08f28d55428a025a03b6d066707793b4	POINT (-122.4105435 47.5705786)	{xmin=-122.410545, xmax=-122.41053, ymin=47.57057, ymax=47.57058}	true

Note that in_bbox is true and is the same part in the WHERE clause.

If I uncomment the bbox tests in the above query then it's missing (as you're pointing out).

But the data is there:

SELECT COUNT(1)
FROM overture.release.v2024_04_16_beta_0
WHERE theme = 'places'
	AND type = 'place'
	AND st_contains(
		st_envelope(
			st_multipoint(
				array [ st_point(-122.44, 47.56),
				st_point(-122.25, 47.71) ]
			)
		),
		st_geomfrombinary(geometry)
	)

has 25903 rows around what you'd expect - the 3/12 release has 26063.

I'm seeing if there's someone we can file a bug or raise an issue. Will keep things updated here.

[danabauer]

Hi @acgeospatial. It looks like this issue has been fixed. I ran the following query in Athena and got 25,901 results.

SELECT
        id,
        names.primary AS primary_name,
        sources[1].dataset AS primary_source,
        ST_GeomFromBinary(geometry) AS geometry
FROM overture.release.v2024_04_16_beta_0
WHERE theme = 'places'
        AND type = 'place'
        AND bbox.xmin > -122.44
        AND bbox.xmax < -122.25
        AND bbox.ymin > 47.56
        AND bbox.ymax < 47.71

Let us know if you're still seeing problems. You can create an issue here and tag it with beta testing.

[mojodna]

For additional context, we think we bumped into an issue with Trino where it doesn't handle Parquet column statistics for floats properly (we switched from storing bounding as doubles to floats in the beta release, saving ~50GB across the whole thing); the results for this bounding box span 2 row groups, and the one that's read contains the 2 results that come back (the one that's skipped contains the other 25k).

The workaround we applied to the public Glue catalog was to tell Athena to ignore statistics (this results in scanning more data; we'll revert it once the underlying issue has been fixed). For people who encounter this with locally-registered tables, run this query to match the published table spec:

ALTER TABLE v2024_04_16_beta_0 
SET TBLPROPERTIES ('parquet.ignore.statistics' = 'true')