NY '19 Fields Differ from Old Data -- Need to clean and process

[trang-van]

@rpasquini , Going through the new NY data I ran into an error about the field names containing "." which Mongo doesn't allow. Looking over the dataset, there are a lot more fields than in the old data (eg. ny_2012.csv). Another issue is that the "created_at" field is a date string, not in Epoch time like the original data. @Emman-Lopez-Oso mentioned that some of the function relies on the Epoch time format.

Currently, I'm working on a copy of the original data where I've cleaned the columns we don't need. I'm working towards a solution to get the new dataset in the same format as the old one, including adding Epoch time column. Before I proceed with uploading in on the "ny" database, I'll need confirmation that the cleaned dataset is what you expected.

[rpasquini]

@trang-van The current project does not have an appropiate documentation on the csv requirements, which is unfortunate. Let me do a first check in order to define the fields the project actually needs and the corresponding formats.
Then you can work on the data to get it into the right formatting (also keeping only the fields we need), or we can work on a population function that solves the problem as it reads the data.