Tests for the image classification and flight tail removal logic

[spwoodcock]

• We just merged in a big update to the imagery upload workflow, including classification into task areas and removal of rejected images based on various criteria.
• @allisonsibrian also did some great work removing flight tails as part of the processing workflow.
• I did some tweaks to all of this / ordering / logic, so ensure it all fits nicely together.
• This needs to be tested thoroughly.
• I have a basic testing dataset: https://github.com/spwoodcock/drone-testdata-agung-1, however this isn't ideal for flight tails testing, as the data is geographically separated and bit random.
• https://github.com/spwoodcock/drone-testdata-agung-2 should be better for flight tail testing. The data need to be synced from the public S3 bucket. I haven't as of 18-01-2025 managed to add the dummy data such as invalid EXIF, blurry images, lens cap simulation, etc - to come!
• The key part of this issue is adding automated tests. We don't actually need to upload imagery for automated tests, instead relying on inserting data into project_images with simulated EXIF data. We can then run the workflows to see if we get the expected sorting of data.

[linear]

TECH-935

[allisonsibrian]

Hi,

Thanks for the invitation to the repo :3! I can self-assign myself to this issue.

I was inspecting the WIP tests and the updated code logic, and never realized how many more cases there were to consider! The addition of segmentation and the vertical takeoff are good additions, so thank you for making the removal more robust!

For this issue, and working on the automated tests:

1. I can create a new test file to mock the Agung-2 failure scenarios (gimbal errors, lens caps, blurry imagery, and invalid coordinates) to verify the ImageClassifier logic.
2. I can add to the flight tail tests to consider for vertical takeoff detection and ensure tails are correctly caught in every segment of a multi-battery batch.
3. I noticed the current WIP tests are failing locally/CI due to ForeignKeyViolation, so I can also fix these issues.

Hoping to tackle this throughout this week, let me know if that sounds like a good flow, or if there is any other specific testing I should account for!

[spwoodcock]

It was actually messing with various test datasets that helped to pick out different scenarios 😄

Your code was so well timed to merge in with the classification logic!

And your plan sounds perfect - appreciate you taking this on so much 🙏

I haven't modified any of the agung-2 imagery yet by the way, so its simply valid imagery from 4 adjacent tasks (all uploaded to public S3), all with flight tails to account for though.

agung-1 has the intentionally introduced errors, like gimbal angles on the horizon and blurry images =)

[allisonsibrian]

I haven't modified any of the agung-2 imagery yet by the way, so its simply valid imagery from 4 adjacent tasks (all uploaded to public S3), all with flight tails to account for though.

agung-1 has the intentionally introduced errors, like gimbal angles on the horizon and blurry images =)

Ah got it, thanks so much Sam!

[spwoodcock]

The tests all work well, except one!

test_multi_batch_rejections

db = <psycopg.AsyncConnection [INTRANS] (host=db user=dtm database=dtm_db) at 0x73292b505c90>, create_test_project = '0d2b0f72-590b-405c-96b7-e5d3925d6d54'
auth_user = AuthUser(id='101039844375937810000', email='admin@hotosm.org', name='admin', profile_img='', role='PROJECT_CREATOR', is_superuser=True)

    @pytest.mark.asyncio
    async def test_multi_batch_rejections(db, create_test_project, auth_user):
        """Verifies that in a batch with multiple segments tails are caught for each flight segment."""
        project_id = uuid.UUID(create_test_project)
        batch_id = uuid.uuid4()
        task_id = uuid.uuid4()
        now = datetime.now(timezone.utc)

        # Project box
        geom = box(115.4, -8.4, 115.6, -8.2)
        outline_wkb = wkblib.dumps(geom, hex=True)

        async with db.cursor() as cur:
            await cur.execute(
                """
                INSERT INTO tasks (
                    id,
                    project_id,
                    project_task_index,
                    outline
                )
                VALUES (
                    %s,
                    %s,
                    %s,
                    %s
                )
                """,
                (task_id, project_id, 1, outline_wkb),
            )

        # Creating two different flight segments separated by a 2 hour gap.
        segments = [
            {"start": now, "label": "flight_A"},
            {"start": now + timedelta(hours=2), "label": "flight_B"},
        ]

        async with db.cursor() as cur:
            for seg in segments:
                for i in range(60):
                    ts = seg["start"] + timedelta(seconds=i * 2)

                    # Creating a path:
                    # 0-5: Vertical and moving up in altitude
                    # 6-12: Moving NORTH (azimuth 0)
                    # 13-60: Mission then moves EAST (azimuth 90)

                    is_vertical = i < 6
                    is_transit = 6 <= i < 13

                    if is_vertical:
                        lat, lon = -8.301, 115.46
                        alt = 100 + (i * 10)
                    elif is_transit:
                        lat, lon = -8.301 + (i * 0.0002), 115.46
                        alt = 200
                    else:
                        lat, lon = -8.301 + (13 * 0.0002), 115.46 + ((i - 13) * 0.0002)
                        alt = 200

                    file_name = f"flight_takeoff_{i:03d}.jpg"
                    s3_key = f"dtm-data/projects/{project_id}/user-uploads/{file_name}"
                    hash_md5 = hashlib.md5(file_name.encode("utf-8")).hexdigest()

                    exif_data = json.dumps(
                        {
                            "DateTimeOriginal": ts.strftime("%Y:%m:%d %H:%M:%S"),
                            "AbsoluteAltitude": str(alt),
                        }
                    )

                    await cur.execute(
                        """
                        INSERT INTO project_images (
                            project_id,
                            filename,
                            s3_key,
                            hash_md5,
                            batch_id,
                            task_id,
                            location,
                            uploaded_by,
                            status,
                            uploaded_at,
                            exif
                        )
                        VALUES (
                            %(project_id)s,
                            %(filename)s,
                            %(s3_key)s,
                            %(hash_md5)s,
                            %(batch_id)s,
                            %(task_id)s,
                            ST_SetSRID(ST_MakePoint(%(lon)s, %(lat)s), 4326),
                            %(uploaded_by)s,
                            'assigned',
                            %(uploaded_at)s,
                            %(exif)s
                        )
                        """,
                        {
                            "project_id": project_id,
                            "filename": file_name,
                            "s3_key": s3_key,
                            "hash_md5": hash_md5,
                            "batch_id": batch_id,
                            "task_id": task_id,
                            "lon": lon,
                            "lat": lat,
                            "uploaded_by": auth_user.id,
                            "uploaded_at": ts,
                            "exif": exif_data,
                        },
                    )
        await db.commit()

        await mark_and_remove_flight_tail_imagery(db, project_id, batch_id, task_id)
        await db.commit()

        # Verify rejection occurs in both flight segments
        async with db.cursor() as cur:
            # Check Flight A rejections.
            await cur.execute(
                "SELECT COUNT(*) FROM project_images WHERE status = 'rejected' AND filename LIKE 'flight_A%%'"
            )
>           assert (await cur.fetchone())[0] >= 5
E           assert 0 >= 5

tests/test_flight_tail_detection.py:674: AssertionError

[allisonsibrian]

Thank you will take a look!