Improve ML model performance analysis workflows

[nathanielrindlaub]

Context

Given that Animl supports using of a growing number of ML models (both object detection models and classification models), and especially now that we've integrated SpeciesNet, we need better, more efficient workflows to try to answer questions such as: What model(s) will work best on my data and for my particular use case? How well has the model I've chosen been working? Are my confidence thresholds optimally set?

These are legitimately hard questions to answer, but Animl should in theory be well suited to addressing them. Users can upload their data, get predictions, correct those predictions to create a ground-truthed labeled dataset, and from there we have everything we need to evaluate model performance (in theory). In practice, I've found this gets fairly complicated and thorny very fast.

Current solution

Currently, if you want to understand how an ML model is performing in production (i.e. on new images, unseen by the model in training or testing), a user would have to:

1. Get predictions: upload a bunch of their images (if it's not already in Animl), with Automation Rules configured to run whatever model(s) with confidence threshold settings they are hoping to evaluate
2. Review and correct those predictions: we need human-verified, ground-truth labels to compare the predictions to, so users must manually review the data using the Animl UI (if they haven't already)
3. Run analysis scripts: ask me to run the analyzeMLObjectLevel.js script (for Object-level performance metrics) or the analyzeMLSequenceLevel.js script (for sequence-level analysis).
4. Interpret results: I would then send over a CSV version of the output to the user. An Excel example of what the output looks like (with some color and formatting) can be found here: sci_biosecurity_2023-4-28--2024-5-29_sequence-level_2024-06-10T1815Z.xlsx. The interpretation of evaluation metrics is also not terribly straight forward, and it's highly use-case dependent (for example the trade off between prioritizing recall vs precision will depend on use case & user's tolerance for missing detections vs dealing with false positives)

This is not a bad start, IMO. The output gives you the number of truePositives, falsePositives, and falseNegatives (the numbers from which all evaluation metrics are derived) as well as the precision, recall, and f1 score (the evaluation metrics) of each TARGET_CLASS (the classes that the ML model predicts). It further breaks out all of these metrics for all of the target classes by Deployment, because model performance can vary widely from deployment to deployment. And you can configure additional filters (project, start date, end date) in the analysis config file.

If your question is, "if a [some species] is in an image, how likely is it that it will be detected and classified correctly?", the scripts also allow you to answer that by performing the analysis at the object level. If your question is "if a [some species] is in an sequence of images, how likely is it that it will be detected and classified correctly in at least one of the images in the sequence?", you can perform the analysis at the sequence/burst level.

Lastly, you can use the scripts to assess JUST the object detection (MegaDetector) step, or and entire ML pipeline (the performance of the object detection and classification steps, together). So if you're interested in assessing how a pipeline is performing end-to-end, i.e. to answer questions like, "If a rodent trips a camera, how likely is it that both my object detector and classifier model will work as expected and I will receive an alert?", the current scripts can answer that.

Limitations and open questions

However, there are also serious limitations and room for improvement to the current workflow. For example:

1. Only works for one model at a time: The workflow described above only produces evaluation metrics for one model pipeline with one particular set of confidence thresholds at a time. If you wanted to compare the performance of two different model pipelines, or if you wanted to compare the performance of the same model but with different confidence threshold settings, you would have to repeat all of the steps above (including the manual prediction review). In an ideal world, given some set of human-labeled images, we could test out a bunch of models at once on that same benchmark dataset without having to re-upload and re-label all of the data each time. The re-labeling part is the most time consuming step and feels pretty redundant if you've already done it once.

2. The scripts were not designed for large classifiers (SpeciesNet) with 1000s of classes at every taxonomic level: when I wrote these scripts our classifiers had at most a couple dozen classes, all at more or less the same taxonomic level, and both how we configure the scripts and how we perform the analysis may have to change to accommodate SpeciesNet. For example, how do we configure the TARGET_CLASSES array in the analysisConfig.js file to scale to 1000s of classes? How do we treat predictions that are close, but not exactly right (e.g. when a "mule deer" is predicted to be a "white-tailed deer") or correct but not specific enough or the correct taxonomic rank (e.g. the model predicts "rodent" but what the user really wants is "brown rat")?

   Key to how we currently deal with some of this is the concept of the predicted class (the class the ML model produces and we're evaluating) and the validation classes (the classes/labels a person may have applied that count as "validating" the predicted class). So if a model's predicted class is "rodent", validation classes might be "mouse", "rat", "rodent", etc. That approach assumed that users would be adding more granular, specific labels that could be considered a validation of higher-level predicted classes, but what happens when the reverse is also possible (SpeciesNet is providing species-level predicted classes like "white tailed deer" and "mule deer" but users are correcting them to "deer"). This may require rethinking and restructuring of our config and analysis scripts.

3. The scripts don't evaluate classifier performance independently: The current scripts are taking some shortcuts and simplifying things a bit. For example, they don't exactly follow the approach you'd use to evaluate an object detector in a model training context, specifically, we aren't evaluating how accurate the location of the bounding boxes are using intersection-over-union or anything like that. Instead, we're ignoring the bounding box accuracy and evaluating it more like a classifier (i.e., if MegaDetector predicted and "animal" and a user either validated it as an "animal" or added some more specific species label on top of it, the scripts would count that as a true positive). Additionally, while the scripts can currently analyze either (a) an object detector independently or (b) the results of using an object detector and a classifier together, they can not analyze the performance of classifier independently. For now I don't think this is a high priority limitation to address, but it's worth being aware of.

4. Is Animl even the right environment to be doing this in? Perhaps a set of offline, local inference scripts are more appropriate?

5. Are these even the right questions to be asking? Should we take more of a UX efficiency/time-saving based approach? Different mistakes by models can slow down manual image review in different ways. For example, a false negative by an object detector means that the user has to stop and draw their own bounding box around a missed object, then add a label to it, which is incredibly slow and much, much slower than invalidating a false positive. Similarly, if users ultimately have to label all deer as "deer", from a time perspective it doesn't matter whether the model predicted "giraffe" or "white tailed deer" or "ungulate" - all would need to be corrected and all would take the same amount of time. That said, there's an argument to be made that close predictions are helpful, even if wrong, because then users can filter on that incorrect class (e.g. "white deer") and bulk-change all of them to "deer" with the bulk-selection tool.

All of that is to say that if the question is "how much time will this model save me", that's different than asking "how accurate is this model", and maybe the time saving question is the more important one.

Resources and prerequisite concepts

If anyone is interested in trying to address these questions and limitations, I'd recommend familiarizing yourself with the following:

• The difference and tradeoffs between precision and recall
• Computer Vision for Ecology lecture on evaluating ML Models
• The types and sequencing of ML models used in a typical camera trap processing pipeline (object detection first, classification second)
• Familiarity with MegaDetector, the most commonly used object detection model for camera trap data, and SpeciesNet, the most common classifier (though Animl supports more than just these two models).
• Animl's data model, specifically the concept of Objects and Labels, Labels' not-validated/validated/invalidated states, Objects' locked/unlocked states, and Images' isReviewed state, and the concept of the "most representative label". I will try to get some better documentation on this together and link it here, but for now there's an example of an Animl image record you can reference below.
• Understand how the current analysis scripts work, specifically the predicted vs validation class concept, and how actuals, true positives, false positives, and false negatives are calculated from Animl data.

// example Animl Image record
// This Image has one Object, which MegaDetector correctly detected and labeled as an "animal"
// and an additional "rodent" Label predicted by the MIRAv2 classifier, which was validated as being correct by a user. 
// In doing so the Object became "locked", the Image became "reviewed" 
// and the "rodent" became the Object's "most representative label"
{
  "_id": "sci_biosecurity:41240fc6d05477747a49c5f1b9bb83c5",
  "bucket": "animl-images-serving-prod",
  "batchId": null,
  "fileTypeExtension": "jpg",
  "dateAdded": {
    "$date": "2025-08-07T12:37:16.565Z"
  },
  "dateTimeOriginal": {
    "$date": "2025-08-07T12:36:07.000Z"
  },
  "timezone": "America/Los_Angeles",
  "make": "BuckEyeCam",
  "cameraId": "X811492D",
  "deploymentId": {
    "$oid": "6164c9f7599481769b50cc2a"
  },
  "projectId": "sci_biosecurity",
  "originalFileName": "p_130372.jpg",
  "imageWidth": 1280,
  "imageHeight": 960,
  "imageBytes": 160599,
  "mimeType": "image/jpeg",
  "userSetData": {
    "TEXT1": "#E5 #G20",
    "TEXT2": "Centinela"
  },
  "model": "X80",
  "location": {
    "_id": {
      "$oid": "68949dfc7ff9639c6068c137"
    },
    "geometry": {
      "type": "Point",
      "coordinates": [
        119.799469444444,
        34.0176694444444
      ],
      "_id": {
        "$oid": "68949dfc7ff9639c6068c138"
      }
    }
  },
  "triggerSource": "Burst #2",
  "tags": [],
  "objects": [
    {
      "_id": {
        "$oid": "68949e007ff9639c6068c15a"
      },
      "bbox": [
        0.4603290557861328,
        0.4360730051994324,
        0.5325396060943604,
        0.5756036639213562
      ],
      "locked": true,
      "labels": [
        {
          "_id": {
            "$oid": "68949e017ff9639c6068c17a"
          },
          "type": "ml",
          "labelId": "rodent",
          "conf": 1,
          "bbox": [
            0.4603290557861328,
            0.4360730051994324,
            0.5325396060943604,
            0.5756036639213562
          ],
          "labeledDate": {
            "$date": "2025-08-07T12:37:21.592Z"
          },
          "mlModel": "mirav2",
          "mlModelVersion": "v2.0",
          "validation": {
            "validated": true,
            "userId": "<email_address>",
            "_id": {
              "$oid": "68949e3620445ceeb29dbc34"
            },
            "validationDate": {
              "$date": "2025-08-07T12:38:14.467Z"
            }
          }
        },
        {
          "_id": {
            "$oid": "68949e007ff9639c6068c158"
          },
          "type": "ml",
          "labelId": "1",
          "conf": 0.9110135436058044,
          "bbox": [
            0.4603290557861328,
            0.4360730051994324,
            0.5325396060943604,
            0.5756036639213562
          ],
          "labeledDate": {
            "$date": "2025-08-07T12:37:20.587Z"
          },
          "mlModel": "megadetector_v5a",
          "mlModelVersion": "v5.0a"
        }
      ]
    }
  ],
  "comments": [],
  "__v": 2,
  "reviewed": true
}

[nathanielrindlaub]

Update

We decided to start by addressing issue No. 2, above: that is, try to adapt the current scripts to work for SpeciesNet. The two main challenges are (1) SpeciesNet has ~2500 classes, so we'll need to automate or at least semi-automate the creation of the analysisConfig.TARGET_CLASS array, and (2) the SpeciesNet classes span all taxonomic levels, so how do we handle situations in which the predicted classes and validating labels are at different taxonomic levels (this issue is described in more detail above).

@lessej did a great job solving the first issue with #350, but we quickly ran into the second when we tested out running the analysis scripts on a palmyra-rat-bucket-test project in which SpeciesNet had (correctly) classified a rat in the images quite a few times, but at different taxonomic levels (rattus species, rodent, muridae family, brown rat), and the reviewer (me) had corrected them all to brown rat. This is problematic because our current scrips were written to evaluate performance on predicted classes for which you might have more than one validating label (the predicted-to-validated label relationship is one-to-many), but not designed to work in situations where the same validating class can validate multiple different predicted classes (the predicted-to-validated label relationship is potentially many-to-one).

The first issue we noticed with the output CSVs (see below) is that the calculation of allActuals is off. The actuals evaluation looks like this:

          // ACTUAL - object must be:
          // (a) locked, (b) has a first valid label that validates the prediction/target class,
          // (i.e., for "rodent" prediction, a firstValidLabel of ["rodent", "mouse, "rat"]),
          if (obj.locked && FVLValidatesPrediction(obj, tClass)) {
            data[key].allActuals++;
          }

...so for each of the predicted classes (rattus species, rodent, etc) that have brown rat as a validation class, our actuals calculation is including all of the examples of the ML model predicting rattus species and a person "validating" it by adding a brown rat label, but also all examples of rodent predictions validated with brown rat too. Basically, we're over counting the actuals, and if your allActuals counts are wrong, the precision, recall, and F1 score will be wrong too. For similar reasons, our falseNegative calculation is misleading as well.

Results of analyzeMLObjectLevel.js on palmyra-rat-bucket-test:
palmyra_bucket_test_speciesnet-classifier_2025-8-1--2025-9-12_object-level_2025-09-12T1850Z.csv
palmyra_bucket_test_speciesnet-classifier_2025-8-1--2025-9-12_object-level_2025-09-12T1850Z_config.json

[nathanielrindlaub]

Taking a step back, I can think of the following different evaluation strategies:

1. strict mode: Evaluate the model's ability to predict the exact class at the correct taxonomic level. So if the predicted class === FVL ("first validated label") exactly, it's a true positive. There's no concept of validating classes.
2. one-to-many validation: Multiple labels/classes can validate a single predicted class. If the FLV is one of the validating classes, it's a true positive. This is how the scripts currently work and is fine for models that predict a flat, non-hierarchical list of classes where with no overlap between them (e.g. the "rodent", "bird", "skunk", "lizard", "fox" classes of the SCI classifier do not overlap).
3. many-to-one validation: A single validating class can validate multiple predicted classes (e.g. "brown rat" can validate "rodent", "brown rat", and "muridae", etc.). This helpful for biosecurity use cases, in which you might want to set an alert for all classes in the muridae family and understand what your recall would be.
4. taxonomic: Take into account the full taxonomic classification of the predicted and validating classes, and generate evaluation metrics at each taxonomic rank. So for each predicted class ("rodent", "brown rat", "muridae", "rattus species") consider an exact match or any child taxa of that class a truePositive. We have taxonomic info for all SpeciesNet classes, but not manually added labels or other models, so this would require the user to add the taxonomic info to each class.

[lessej]

This is a good list of strategies. As much as it would be nice to pick one and run with it, I think we'll likely benefit from supporting all of them in some capacity. I am adding some additional thoughts to hash out when discussing how to move forward:

• How much of the overlapping classes is down to a usage issue? For example, knowing that rattus is a subset of muridae, should both of these labels be used as project labels at the same time? When adding project labels and labelling data, should users (us) be conscious about adding and validating labels that overlap with each other?
• R.E many-to-one validation: we probably want to split this into two types:
  • no overlap in predicted labels: there's no way we can group / roll up the predicted labels. I don't know what use there would be for this and maybe this is a boundary we can define.
  • predicted labels with overlap: depending on how the data is structured, there's some way we can make informed decisions about how to roll up / combine predicted labels into groups. In this case, I think we approach it either as a 'predicted group' (instead of single predicted label) or if the labels are structure in an increasingly specific way, we can take the approach of analyzing at each level of specificity.
• taxonomic: I feel that this is a sub-scenario of the earlier many-to-one bullet point. Fortunately, I think it's also one of the more well-defined cases which is on the easier end of solving. For speciesnet, it seems worth it to use the taxonomic data but I am more hesitant about trying to push the taxonomic structure onto other classifiers which don't already use it. Some questions to get us started on this path: how far up the taxonomic tree does speciesnet label? Ex., does it attach 'mammal' or does it start attaching labels at a more specific rank? How do we present this information to users?

[nathanielrindlaub]

How much of the overlapping classes is down to a usage issue?

SpeciesNet produces classes at all taxonomic levels, and if an ML model produces a label for a user's data we automatically add it to their Project Labels, so I don't think it's a usage issue.

R.E many-to-one validation: we probably want to split this into two types

Let's discuss this, I'm not 100% sure I am tracking.

For speciesnet, it seems worth it to use the taxonomic data but I am more hesitant about trying to push the taxonomic structure onto other classifiers which don't already use it...how far up the taxonomic tree does speciesnet label?

I agree, but I think I'm warming up to taking the taxonomic approach, and if I'm not mistaken I think it should essentially solve the many-to-one scenario. The rough process would be:

1. create a list of Project Labels that were produced by the ML model you're evaluating
2. Look up the corresponding ML model categories and check whether they have taxonomy data.
3. For any that don't have taxonomy data, prompt the user to input it somehow
4. Iterate over all images that have predictions of those classes and generate a set of all First Validated Labels used to validate images with those classes.
5. Again, if any of those FVLs don't have taxonomic data, prompt the user to supply it
6. At this point the script should be good to go: it would then count up the actuals, truePositives, falsePostives, falseNegatives for each of those predicted labels, where:
   • an actual of a given predicted class would be any object who's FVL is an exact match or child taxa of the class, regardless of whether the predicted class was actually predicted for that object or not.
   • a truePositive of a given predicted class would be any object that both (a) received a prediction of that class (or child taxa of the class) and (b) who's FVL is an exact match or child taxa of the predicted class.
   • a falsePositive of a given predicted class would be any object that both (a) received a prediction of that class (or child taxa of the class) and (b) who's FVL is NOT an exact match nor child taxa of the predicted class.
   • a falseNegative of a given predicted class would be any object that (a) didn't receive a prediction of that class (or any child taxa of that class) but (b) who's FVL is an exact match or child taxa of the predicted class.

I am not super concerned about adding the taxonomic data to the classifiers that don't have it. It would be a bit of a chore to add them but realistically wouldn't take more than an hour or two. Adding taxonomic data for user-created Labels would be trickier, but I can picture a couple UI solutions for it.