UNITAC Contract: Improving the photo upload experience & other tasks

[spwoodcock]

Overall milestone: https://github.com/hotosm/drone-tm/milestone/1

Pt1: improvements to the photo uploading process

Summary: refactors the current photo upload process making it more scalable, improves the user experience of photo uploads on bad internet connections.

#540
#553
#636

Current workflow

• Currently users lock each task, generate flightplans, and have the captured imagery on their SD Card in the drone.
• They return to base, and then need to separate out which images were for each task, and upload the imagery to each.
• It's down to the user to ensure they upload the correct imagery for the correct task.
• The imagery files are stored in the S3 bucket under dtm-data/projects/{project_uuid}/{task_uuid}/images/{filename}.jpg

Proposed workflow

• The user returns to base with an SD Card full of imagery, and accesses the project on DroneTM.
• They don't need to work out which imagery is for what task. Instead on the project page we have a single 'Upload imagery' place (i.e. per project, not per task).
• The user dumps their imagery into this dropzone or bulk file selector, we upload via Uppy / multipart, then sort the imagery into task using EXIF.
  • The photos land in a staging directory: dtm-data/projects/{project_uuid}/user-uploads/{filename}.jpg
  • On a schedule for the backend we can sort this imagery into the correct task location using it's EXIF tag geolocation (intersect with the task areas to find which one).
  • Currently we extract the EXIF into a images.json file. This definitely needs improvement, see Replace images.json approach as it's not scalable, not transactional, and is fragile #636.
  • Once the task is determined, we do an S3 file move from user-uploads to dtm-data/projects/{project_uuid}/{determined_task_uuid}/{filename}.jpg
  • Uploading the same photo twice shouldn't be a problem: we simply replace what is currently in the S3, or ignore it - idempotent workflow. We could determine if it's a dup using a stored file hash.
• We the user visits a task page again, we can display all of the uploaded photo locations on the map, so they can visually inspect for gaps.
  • If happy with the photo locations / included photos, the user clicks to process the 'fast ortho' for the task area only.
  • The user should be able to click on a photo and load the photo from the S3 via pre-signed URL inside a popup / dialog.
  • From the dialog, the user can click a button to delete the image from the S3 if they don't want to include it / something wrong with the location or image.
  • Note we already have a small UI in place to do some of this, but it should be refined.

Note

We could consider reading the EXIF info prior to upload using a js lib like https://github.com/mattiasw/ExifReader

But we would also need to do the photo geolocation / task intersection calculation - probably best handled on the backend.
This does provide a slight delay in the processing, but it simplifies the frontend, and makes it much easier to re-run sorting and debug from the server-side.

Implementation

• Described in linked issue, using Uppy / multipart S3.
• Let's see what the default Uppy dropzone is like - perhaps we use that.
• Worst case, we wrap the Uppy code in our own WebAwesome webcomponent in our component library: https://github.com/hotosm/ui
• Note that any designed UI elements should ideally use WebAwesome / HOTOSM UI where possible🙏 (or at the very least follow the design guide https://github.com/hotosm/ui-design

Triggering the photo classification

• Links to pt3 below where we can hook into this workflow to better sort through bad imagery.
• But for now, we simply need a way to:
  1. Endpoint /task_id/uploads/status for how many photos are remaining to be sorted / classified. The status could be displayed to the user on the task page.
  2. Endpoint to trigger the photo classification. Button displayed on the task page if there are still photos to classify - triggers the process.
  3. Cronjob running every 30 minutes to do run classification / sorting code automatically.

Pt2: better feedback for processing workflows

Summary: give a bit more feedback to users on the status of their processing, and also allow them to retry failed processing.

#603
#515

• Once processing is done for a task, perhaps the user wants to refly the task, or they missed some photos on the original upload, etc.
  • We provide the option to either (1) wipe all existing photos and start again (2) append new photos to the task.
  • The user can then use the newly uploaded photos to re-run the processing and get a new 'fast ortho' output.

Pt3: filtering of bad imagery before processing

Summary: during photo classification step, add some logic to scan the imagery and remove 'bad' data, such as photos pointing to the sky / horizon, blurry photos, photos taken too and from the takeoff point (discard).

#541 (server side)
#511 (client side)
#599 (server side)

Note

All of this validation is done on the backend. If we host the backend and S3 in the same AWS zone, this shouldn't be a huge deal (we have to download the imagery to backend for checking).

However, it would be really nice to do some basic EXIF validation on the frontend (and perhaps in the QGIS plugin in the future).

1. Check if EXIF tags are present (coords, timestamp, camera make and modal)
2. (Optional): check if the EXIF lat/lon fall in the project area
3. Gimbal angle in EXIF (if present), is >-30, otherwise probably discard as not useful (should be present for DJI as GimbalPitchDegree, GimbalRollDegree, but other manufacturers may use other EXIF tags, so keep this in mind).

This would reduce bandwidth from user --> server and be a bit more efficient.

Pt4: post-upload / processing workflows

Summary: display the processed task 'fast orthophoto' to the user, then allow the user to tag areas that are not well covered for a second flightplan and re-run either (1) all of the task (2) part of the task.

Manual: #512
Automated: #513

[spwoodcock]

Link to architecture / flow diagrams: https://gist.github.com/FeezyHendrix/db2805a985ab0ee36715960df1e3cea2

[spwoodcock]

The contract is coming to a close.

Summary of what we achieved

1. Removed flakey images.json writing from S3 (Replace images.json approach as it's not scalable, not transactional, and is fragile #636)

   • Replaced with database records per image
   • Transactional / handles multiple users uploading at the same time

2. Moved the upload from per-task to per-project

   • Now a single place to dump files, which get sorted into tasks on the backend
   • The user can return from the field with an SD card and upload all the images without sorting (Add a second workflow for drone imagery upload: photo dump per SD card #540)

3. Implemented multipart upload, so connections can break and uploads are resumed

   • Improve imagery upload workflow and provide fallback for poor internet connections #553
   • Let's do some good testing and refinement of this if possible, before closing the issue.

4. Implemented better user feedback on image upload:

   • Photos are analysed and sorted into tasks
   • Images falling outside all tasks are flagged to not be included
   • Images with issues: gimbal angle wrong, lens cap on, etc are flagged to not be included (Remove images with a camera gimbal pitch greater than theshhold #511 & Pre-process imagery and flag potentially 'bad' photos per SD-card dump, not per task #541)
   • A volunteer @allisonsibrian is working on Remove flightplan tails during post processing #599 I think.

Potential things to continue (outside the contract)

Processing feedback and restart:

• Poll and display processing progress in a dialog: The progress of imagery processing should be polled and displayed in a dialog #603
• Allow re-running of processing if new imagery is uploaded: Provide option to re-run processing after new imagery is uploaded #515

Detecting coverage gaps from the imagery uploaded:

• Manual: Web UI to allow user to select areas from the first pass ortho that were missed by flights & create a second pass flight #512
• Automated: Automatically detect if there are gaps in the first pass ortho and allow the user to create a second-pass flight plan #513

[spwoodcock]

Creating a test dataset:

1. Download a directory of images from drone flights (ideally all in same area).
2. Load into QGIS and create a bounding box geojson (your new AOI).
3. View EXIF data

exiftool -csv \
    -FileName \
    -DateTimeOriginal \
    -GPSLatitude \
    -GPSLongitude \
    -AbsoluteAltitude \
    -GimbalPitchDegree \
    -CameraPitch \
    -FlightYawDegree \
    ./ > image_metadata.csv

4. Make a few file duplicates:

• Name a couple _DUP.JPG to denote duplicates. No additional modification neede.
• Name a couple GIMBAL_HORIZON.JPG, then set their gimbal pitch and degree to point upwards:

  exiftool -overwrite_original \
    -GimbalPitchDegree=0 \
    -CameraPitch=0 \
    *_GIMBAL_HORIZON.JPG
  

• Name a couple _GIMBAL_UP.JPG, then set their gimbal angle to 0 (horizon shot):

  exiftool -overwrite_original \
    -GimbalPitchDegree=+30 \
    -CameraPitch=+30 \
    *_GIMBAL_UP.JPG
  

• Name a couple _FAR_AWAY.JPG, then:

  exiftool -overwrite_original \
    -GPSLatitude=48.8566 \
    -GPSLatitudeRef=N \
    -GPSLongitude=2.3522 \
    -GPSLongitudeRef=E \
    *_FAR_AWAY.JPG
  

• Name a couple _WRONG_HEMISPHERE.JPG, then:

  exiftool -overwrite_original \
    -GPSLatitude=8.29 \
    -GPSLatitudeRef=N \
    -GPSLongitude=115.45 \
    -GPSLongitudeRef=W \
    *_WRONG_HEMISPHERE.JPG
  

• Name a couple _INVALID_COORD.JPG, then:

  exiftool -overwrite_original \
    -GPSLatitude=250.0 \
    -GPSLatitudeRef=N \
    -GPSLongitude=325 \
    -GPSLongitudeRef=W \
    *_INVALID_COORD.JPG
  

• Name a couple _LENS_CAP.JPG, then make a couple of adjustments to the coords:

  exiftool -overwrite_original \
          -GPSLatitude="-8.303950" \
          -GPSLongitude="115.484150" \
          -GPSLatitudeRef=S \
          -GPSLongitudeRef=E \
          DJI_20251001213509_0003_D_LENS_CAP.JPG
  
  exiftool -overwrite_original \
    -GPSLatitude="-8.241715" \
    -GPSLongitude="115.494470" \
    -GPSLatitudeRef=S \
    -GPSLongitudeRef=E \
    DJI_20251027143608_0008_D_LENS_CAP.JPG
  

  Then modify these images with your image editor of choice to be mostly black images (signalling it's a lens cap on).
  If the EXIF gets trashed doing this, you can copy EXIF from the original:

  exiftool -overwrite_original \
    -TagsFromFile DJI_20251001213509_0003_D.JPG \
    -AbsoluteAltitude \
    -GimbalPitchDegree \
    -FlightYawDegree \
    DJI_20251001213509_0003_D_LENS_CAP.JPG
  

  Then modify the coords again.
• Name a couple _POOR_SHARPNESS.JPG, then modify the sharpness with a simple blur filter and:

  exiftool -overwrite_original \
    -GPSLatitude="-8.303960" \
    -GPSLongitude="115.484160" \
    -GPSLatitudeRef=S \
    -GPSLongitudeRef=E \
    *_POOR_SHARPNESS.JPG
  

Note

Errors like
Warning: [minor] Possibly incorrect maker notes offsets (fix by -456?) - DJI_20251027143608_0008_D_LENS_CAP.JPG
Are absolutely fine. They reference proprietary MakerNotes, which we don't need.

[spwoodcock]

Feedback 15-01-2025

1. Move the button to upload the modal #717

The button should be more prominent.
It should be larger and very clear to the user.
Centralise with the sidebar UI on the left.

2. What if the user aborts the process? ✅ (I think is isn't actually an issue)

• If the user is part way through uploading, then clicks to cancel, we delete the batch of images from the db.
• What if they close their browser, leave it hanging and return in a few days, something else?
• I think if they try to upload again, it will be marked as duplicate on return right? Needs to be thoroughly tested in a few scenarios.
• See the scenario in 3. below: I didn't complete the process, then tried to upload again, but they are all marked as dups. How can we continue to the task review and final steps for processing now?

3. Scenario when all images are dups #716

• If the user uploads images, then returns, then tries to upload again. Or if they accidentally select images they have already uploaded.
• They are correctly identified as duplicates. But the next page fails to render the map as there are no images points to load on it. Perhaps in this scenario, we should force abort the process instead of continuing to the next page?

4. User error: can continue before batch processed #717

• It's possible to click continue before the whole batch is processed.
• We should definitely only enable the continue button after they are all done.
• When I continue, then I can't go back, as it makes me abort the process. I don't think all the images were processed when I click continue though.
• I actually think this is a problem on all the pages? We shouldn't be able to continue unless the step is complete. For example, it's possible to continue before the classification is triggered.

5. The map during task validation step should display all the images from all tasks, but does not. #715

(edit: perhaps the reason for this is because some of the images have invalid coordinates, and some are the other side of the planet. The map can't display those, so they should be filtered out before rending this map)

• I uploaded 43 images, a few dups. Perhaps none of them were correctly classified into tasks?
• In this scenario, what do we show the user?
• I think we should be showing the images that were not classified into task areas (Images not matching tasks), for clarity here (inform the user).
• Updated I also tested this on the dev server, and the map never populates / stays blank?

6. Logic to handle additional validation metrics

I'll take a look at these:

• Lens cap still on: black image.
• Overly exposed photos - sun shots?
• Sky ratio and horizon detection - do we need this if we use gimbal angle?

7. Check the multipart pause and cancel is tested well

• What happens if we abort upload while its happening?
• Can we pause and resume without issues?

8. Conflicting popup message for user

• When clicking 'Classify' we get a green popup to say the classification was successful.
• I think this is potentially because of some logic

[spwoodcock]

Feedback 16/01/2026

Captured in issue #714

1. Issue with processing triggering - important!

• I selected 1/4 tasks to process and clicked 'Process'
• Polling began for a the single task, as expected.
• After some time, perhaps a minute, the next poll suggested that all 4/4 tasks are now submitted for processing, when I didn't specify that.

2. How to view task processing status after closing dialog

• If I click 'Finish' at the end, the dialog closes that does the status polling.
• At this point the page should probably refresh anyway, to give the updated PROCESSING_STARTED status / colour to the tasks on the map.
• But from here, how does the user track the processing status and/or receive feedback about the status? Should they just wait until the task changes colour on the map on refresh?
• We probably need a 'Processing Status' dialog or something, to view the status of the processing for each task.
• Within this dialog, we also need a way to trigger the processing for the entire project (once all tasks are processed individually). Currently we just have a 'Start Processing' button on the project page. This button should be moved inside the proposed dialog, and contextualised to differentiate the individual task processing vs the entire project processing. What will each type of processing produce (fast ortho vs final stitched ortho product + 3d model), and the fact the final processing may take more time.

[spwoodcock]

Feedback 18/01/2026

1. Display rejected images on the task review map #701

We currently separate out 'rejected' images from task-assigned images:

• It might actually be good to instead move the images that were rejected, but still assigned a task, into the task dropdown. They should be indicated as rejected clearly with the existing 'Rejected' text written on them.
• They should also display on the map as red dots, instead of green. This allows the user to clearly all the uploaded images for a task, but which ones were accepted / rejected.
• It would be really cool to have a toggle on the map, where we can 'Hide rejected photos', only viewing the green accepted photos.
• Those rejected because they are not assigned to a task, should be shown in an 'Unassigned images' section at the bottom (replacing the 'Rejected images' section).

[spwoodcock]

Uploader test on prod

I conducted a pretty good test of new functionality using this test dataset

The uploading of 1000 files went very smoothly (albeit on a good connection)

I switched down to lower upload speed connections during the process, to simulate connection speed fluctuations - 15Mbps (Wifi) --> 3 Mbps (4G) --> 1Mbps (3G). Nothing failed while doing this - the estimated time to complete just increased as expected 🙌

Image classification test on prod

TBC