Week 4 Presentation Proposal - Data Set Anomaly Detection using the k-NN Algorithm

[pilsnerfrajz]

Assignment Proposal

Title

Data Validation with TensorFlow and k-NN

Names and KTH ID

• William Hedenskog (whed@kth.se)
• Joakim Sundman (jsundma@kth.se)

Deadline

• Week 4

Category

• Presentation

Description

We plan on presenting two methods for data validation of a data set. One will be based on the k-NN algorithm and the other will use the tool TensorFlow for anomaly detection. The purpose of data validation is to remove data that is not suitable for its intended use case.

We will explain how the methods work and compare them in terms of performance and simplicity.

Relevance

Data validation is an important step in AIOps, to ensure high-quality data. Failing to detect data anomalies could have a large negative impact on model performance. It is easier to prevent tainting an AI model with bad data, than to remove it after it has already been integrated with the model.

[ericcornelissen]

The proposal looks interesting. Could you clarify where you got the idea from/what this is based on?

[pilsnerfrajz]

The proposal looks interesting. Could you clarify where you got the idea from/what this is based on?

So the algorithm is mentioned in suggested reading Building Machine Learning Models Like Open Source Software. Then while reading about the benefits of AIOps etc., we found Anomaly Detection to be on many lists, e.g. this one. We thought we could combine these into a CI workflow as per the proposal.

Does this answer your question @ericcornelissen ?

[sofiabobadilla]

@pilsnerfrajz
k-nn , while simple, can certainly help with anomaly detection. However, I’m not fully convinced that the scenario you are proposing is realistic—particularly the idea that “the test should/could trigger each time new data or tuning data is added to the database as part of a CI pipeline.”

Instead, I would suggest focusing your presentation on metrics and methods for data validation, and perhaps including a comparison of existing AIOps tools such as: TensorFlow Data Validation, Great Expectations, or Evidently AI

Do ping me once you reframe the proposal :)

[pilsnerfrajz]

Is the algorithm or the process itself not realistic @sofiabobadilla ? We are thinking of a scenario where data has been pre-processed and then as an extra check we run this particular test against the data to protect the model. This test could be generalized to some other algorithm or some tool like TensorFlow. Since the presentation needs a technical part, we thought we could go into detail on how k-NN works, but the main idea is the implementation of an automated check before data is merged.

If it is the algorithm that is too simple/limited, we could add another method for comparison or similar.

Still, if you think it is completely unreasonable, we will of course change the proposal 😄

[JoakimSundman]

Are we gonna get anymore feedback? We don't have much time to prepare a presentation anymore so we would be happy if we could get confirmation asap. @sofiabobadilla @ericcornelissen

[pilsnerfrajz]

Since we have not received any feedback, we have this draft proposal as a backup. Let us know if this is more suitable @ericcornelissen @sofiabobadilla !

Data validation with TensorFlow

TensorFlow can be used to detect missing data points and data that does not conform to a user-specified schema. If the data is expected to be within a certain threshold, anomalies can be detected in this step. Compared to an algorithm like k-NN, this is much more robust in detecting various types of data anomalies, but requires more preparation by the user. k-NN on the other hand does not need a schema, but the tradeoff is that it is computationally heavy.

Data validation is an important step in AIOps, to ensure high-quality data. Failing to detect data anomalies could have a large negative impact on model performance.

[sofiabobadilla]

@pilsnerfrajz @JoakimSundman

I like the new proposal

Relevance and difficulty is great, the description is a bit high level. In the presentation we will expect you to understand how the tool works and detects the anomalies (does not mean to learn every single algorithm, but know which one they use and how)

Please update title, description and README (asap) so I can merge.

Looking forward to your presentation.

[pilsnerfrajz]

@sofiabobadilla We good now? 🥳 It is still a bit high level, but it is hard to be more specific right now 🤷‍♂️

[sofiabobadilla]

Yes it is good, merging 🤠