Skip to content

feat: add adjustable start and end for analysis window #346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lessej wants to merge 10 commits into
base: main
Choose a base branch
from
Open

feat: add adjustable start and end for analysis window #346

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c902e74
feat: add adjustable start and end for analysis window
lessej Aug 15, 2025
070bb24
fix: use indexes to aid in slicing
lessej Aug 15, 2025
55109fb
fix: refactoring to use mongo queries
lessej Aug 26, 2025
05a5e86
fix: use db queries instead of in memory filtering
lessej Aug 27, 2025
5f64577
fix: make linter pass
lessej Aug 27, 2025
a3a2915
fix: move window adjustment to make it more clear what window it's using
lessej Aug 27, 2025
f5d6b50
fix: fix linting errors
lessej Aug 27, 2025
74f0915
fix: revert back to for await const
lessej Sep 4, 2025
47902cf
fix: ajust in both directions from start/end of project
lessej Sep 4, 2025
b8bdef0
feat: update adjustable window in sequence level
lessej Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions src/scripts/analysisConfig.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,19 @@ const analysisConfig = {
PROJECT_ID: 'sci_biosecurity',
START_DATE: '2023-4-28',
END_DATE: '2024-5-29',

// Attempt to find a start and end date for automation rule applications.
// Starts at earliest image and looks for the first instance of an object
// with a label from the target ML model.
// Repeats the process in the reverse direction to find the latest image.
//
// Use this option if you do not know when the automation rule applied
// and want to ensure you're analyzing all images that have been processed
// by your target model.
//
// This can override START_DATE and END_DATE
AUTO_ADJUST_TIME_WINDOW: true,

ML_MODEL: 'mirav2', // first use of 'mirav2' was 2023-4-28
TARGET_CLASSES: [
// class naming convention: '<label.name>:<label._id>'
Expand Down
81 changes: 77 additions & 4 deletions src/scripts/analyzeMLObjectLevel.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import { ProjectModel } from '../../.build/api/db/models/Project.js';
* STAGE=prod AWS_PROFILE=animl REGION=us-west-2 node ./src/scripts/analyzeMLObjectLevel.js
*/

const { ANALYSIS_DIR, PROJECT_ID, START_DATE, END_DATE, ML_MODEL } = analysisConfig;
const { AUTO_ADJUST_TIME_WINDOW, ANALYSIS_DIR, PROJECT_ID, START_DATE, END_DATE, ML_MODEL } = analysisConfig;

const TARGET_CLASSES = analysisConfig.TARGET_CLASSES.map((tc) => ({
predicted_id: tc.predicted.split(':')[1],
Expand Down Expand Up @@ -138,17 +138,89 @@ function FVLValidatesPrediction(obj, tClass) {
}
}

async function tryAdjustAutomationWindow() {

@nathanielrindlaub nathanielrindlaub Sep 4, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems solid for a first pass at a solution, and I don't want to over think it, but there are circumstances when users use model A for some period of time then try out model B then decide A was a better option so the switch back to A. This solution doesn't account for that of course but I think that's ok for now.

console.log('attempting to adjust automation window...');
const imageQuery = {
$match: {
projectId: PROJECT_ID,
reviewed: true,
objects: {
$elemMatch: {
labels: {
$elemMatch: {
mlModel: ML_MODEL
}
}
}
}
},
};
const firstMlLabelAfterStart = await Image.aggregate([
imageQuery,
{ $sort: { dateAdded: 1 } },
{ $limit: 1 }
]);
const lastMlLabelAfterStart = await Image.aggregate([
imageQuery,
{ $sort: { dateAdded: -1 } },
{ $limit: 1 }
]);

if (
!firstMlLabelAfterStart ||
firstMlLabelAfterStart.length < 1 ||
!lastMlLabelAfterStart ||
lastMlLabelAfterStart.length < 1
) {
throw new Error('unable to find a valid first and last image in automation window.');
}

const dateOfFirstMlLabelAfterStart = new Date(firstMlLabelAfterStart[0].dateAdded);
dateOfFirstMlLabelAfterStart.setDate(dateOfFirstMlLabelAfterStart.getDate() + 1);

const dateOfLastMlLabelAfterStart = new Date(lastMlLabelAfterStart[0].dateAdded);
dateOfLastMlLabelAfterStart.setDate(dateOfLastMlLabelAfterStart.getDate() - 1);

const newStart = dateOfFirstMlLabelAfterStart.toDateString() !== (new Date(START_DATE)).toDateString()
? dateOfFirstMlLabelAfterStart.toISOString().split('T')[0]
: undefined;

const newEnd = dateOfLastMlLabelAfterStart.toDateString() !== (new Date(END_DATE)).toDateString()
? dateOfLastMlLabelAfterStart.toISOString().split('T')[0]
: undefined;

return {
newStart: newStart,
newEnd: newEnd
};
}

// main function
async function analyze() {
let startDate = START_DATE;
let endDate = END_DATE;
console.log(
`Analyzing ${ML_MODEL} performance in ${PROJECT_ID} Project between ${START_DATE} and ${END_DATE}...`,
`Analyzing ${ML_MODEL} performance in ${PROJECT_ID} Project between ${startDate} and ${endDate}...`,
);
console.log('Getting config...');
const config = await getConfig();
console.log('Connecting to db...');
const dbClient = await connectToDatabase(config);

try {
// adjust analysis window to try and avoid false negatives
if (AUTO_ADJUST_TIME_WINDOW) {
const { newStart, newEnd } = await tryAdjustAutomationWindow();
if (newStart) {
console.log(`found a more likely start to the automation window: ${newStart}`);
}
if (newEnd) {
console.log(`found a more likely end to the automation window: ${newEnd}`);
}
startDate = newStart ?? startDate;
endDate = newEnd ?? endDate;
}

// set up data structure to hold results
const project = await ProjectModel.queryById(PROJECT_ID);
const cameraConfigs = project.cameraConfigs;
Expand Down Expand Up @@ -181,7 +253,7 @@ async function analyze() {
fs.mkdirSync(analysisPath, { recursive: true });
}

const root = `${PROJECT_ID}_${ML_MODEL}_${START_DATE}--${END_DATE}_object-level_${dt}`;
const root = `${PROJECT_ID}_${ML_MODEL}_${startDate}--${endDate}_object-level_${dt}`;
await writeConfigToFile(root, analysisPath, analysisConfig);

const csvFilename = path.join(analysisPath, `${root}.csv`);
Expand All @@ -190,9 +262,10 @@ async function analyze() {
stringifier.on('error', (err) => console.error(err.message));

// stream in images from MongoDB
const aggPipeline = buildBasePipeline(PROJECT_ID, START_DATE, END_DATE);
const aggPipeline = buildBasePipeline(PROJECT_ID, startDate, endDate);
const imgCount = await getCount(aggPipeline);
console.log('image count: ', imgCount);

const progress = new cliProgress.SingleBar({}, cliProgress.Presets.shades_classic);
progress.start(imgCount, 0);

Expand Down
80 changes: 76 additions & 4 deletions src/scripts/analyzeMLSequenceLevel.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import cliProgress from 'cli-progress';
* STAGE=prod AWS_PROFILE=animl REGION=us-west-2 node ./src/scripts/analyzeMLSequenceLevel.js
*/

const { ANALYSIS_DIR, PROJECT_ID, START_DATE, END_DATE, ML_MODEL, MAX_SEQUENCE_DELTA } =
const { AUTO_ADJUST_TIME_WINDOW, ANALYSIS_DIR, PROJECT_ID, START_DATE, END_DATE, ML_MODEL, MAX_SEQUENCE_DELTA } =
analysisConfig;

const TARGET_CLASSES = analysisConfig.TARGET_CLASSES.map((tc) => ({
Expand Down Expand Up @@ -203,17 +203,89 @@ function processSequence(sequence, deployment, data) {
return data;
}

async function tryAdjustAutomationWindow() {
console.log('attempting to adjust automation window...');
const imageQuery = {
$match: {
projectId: PROJECT_ID,
reviewed: true,
objects: {
$elemMatch: {
labels: {
$elemMatch: {
mlModel: ML_MODEL
}
}
}
}
},
};
const firstMlLabelAfterStart = await Image.aggregate([
imageQuery,
{ $sort: { dateAdded: 1 } },
{ $limit: 1 }
]);
const lastMlLabelAfterStart = await Image.aggregate([
imageQuery,
{ $sort: { dateAdded: -1 } },
{ $limit: 1 }
]);

if (
!firstMlLabelAfterStart ||
firstMlLabelAfterStart.length < 1 ||
!lastMlLabelAfterStart ||
lastMlLabelAfterStart.length < 1
) {
throw new Error('unable to find a valid first and last image in automation window.');
}

const dateOfFirstMlLabelAfterStart = new Date(firstMlLabelAfterStart[0].dateAdded);
dateOfFirstMlLabelAfterStart.setDate(dateOfFirstMlLabelAfterStart.getDate() + 1);

const dateOfLastMlLabelAfterStart = new Date(lastMlLabelAfterStart[0].dateAdded);
dateOfLastMlLabelAfterStart.setDate(dateOfLastMlLabelAfterStart.getDate() - 1);

const newStart = dateOfFirstMlLabelAfterStart.toDateString() !== (new Date(START_DATE)).toDateString()
? dateOfFirstMlLabelAfterStart.toISOString().split('T')[0]
: undefined;

const newEnd = dateOfLastMlLabelAfterStart.toDateString() !== (new Date(END_DATE)).toDateString()
? dateOfLastMlLabelAfterStart.toISOString().split('T')[0]
: undefined;

return {
newStart: newStart,
newEnd: newEnd
};
}

// main function
async function analyze() {
let startDate = START_DATE;
let endDate = END_DATE;
console.log(
`Analyzing ${ML_MODEL} performance in ${PROJECT_ID} Project between ${START_DATE} and ${END_DATE} at the sequence level...`,
`Analyzing ${ML_MODEL} performance in ${PROJECT_ID} Project between ${startDate} and ${endDate} at the sequence level...`,
);
console.log('Getting config...');
const config = await getConfig();
console.log('Connecting to db...');
const dbClient = await connectToDatabase(config);

try {
// adjust analysis window to try and avoid excessive false negatives
if (AUTO_ADJUST_TIME_WINDOW) {
const { newStart, newEnd } = await tryAdjustAutomationWindow();
if (newStart) {
console.log(`found a more likely start to the automation window: ${newStart}`);
}
if (newEnd) {
console.log(`found a more likely end to the automation window: ${newEnd}`);
}
startDate = newStart ?? startDate;
endDate = newEnd ?? endDate;
}

// set up data structure to hold results
const project = await ProjectModel.queryById(PROJECT_ID);
const cameraConfigs = project.cameraConfigs;
Expand Down Expand Up @@ -248,7 +320,7 @@ async function analyze() {
fs.mkdirSync(analysisPath, { recursive: true });
}

const root = `${PROJECT_ID}_${ML_MODEL}_${START_DATE}--${END_DATE}_sequence-level_${dt}`;
const root = `${PROJECT_ID}_${ML_MODEL}_${startDate}--${endDate}_sequence-level_${dt}`;
await writeConfigToFile(root, analysisPath, analysisConfig);

const csvFilename = path.join(analysisPath, `${root}.csv`);
Expand All @@ -257,7 +329,7 @@ async function analyze() {
stringifier.on('error', (err) => console.error(err.message));

// get image count
const aggPipeline = buildBasePipeline(PROJECT_ID, START_DATE, END_DATE);
const aggPipeline = buildBasePipeline(PROJECT_ID, startDate, endDate);
const imgCount = await getCount(aggPipeline);
console.log('image count: ', imgCount);
const progress = new cliProgress.SingleBar({}, cliProgress.Presets.shades_classic);
Expand Down
Loading