scripts for data fusion

Anaconda with python 3 is recommanded for running python scripts
To dump the data, please use valentino and the 'export to csv' option. The order of the columns should be the same as that in the database.

runing order:
python:
main.py

matlab:
importcsv_and_7
importcsv_ios_7
upsample_and_8
upsample_ios_8
gpsExtracion_main_9

steps before running main.py
1, dump location data from database as location.csv
2, dump activity data from database as activity.csv
3, dump wifi data from database as wifi.csv
4, dump phq9 data from database as phq9.csv
5, dump user mapping data (depressed) from database as mapping.csv

to split big csv file into small files based on uid

'1) split_activity.py'
'1) split_location.py'
'1) split_wifi.py'
input: activity.csv, location.csv, wifi.csv
output: uid*.csv in sub-folders location: android/uid*.csv & ios/uid*.csv
wifi: ios_wifiraw/uid*.csv & android_wifiraw/uid*.csv
actiity:

deduplication

'2) removedup.py'
input: uid*.csv with duplications
output: uid*.csv w/o duplications
location: ios_remdup/uid*.csv & android_remdup/uid*.csv
wifi: ios_wifiraw_remdup/uid*.csv & android_wifiraw_remdup/uid*.csv

look for a corresponding latitude and longitude match in location data at a given timestamp for a given BSSID in wifi data.

'3) mapgpsdata.py'
input: location/uid*.csv, wifi/uid*.csv
output: matchedgps_androidcheck.csv, matchedgps_ioscheck.csv

merge mapping files from ios and android

'4) mergecsv.py'
input: matchedgps_androidcheck.csv, matchedgps_ioscheck.csv
output: matchedgps_final.csv

cauculate the GPS location for BSSID

'5) calculatemedian_latlong.py'
input: matchedgps_final.csv
output: bssidmappingfile.csv

replace the BSSID in the raw wifi file with coordinates

'6) processwifi.py'
input: bssidmappingfile.csv, wifi/uid*.csv
output: wifiprocess/uid*.csv

data fusion - merge data from diffrernt sources

importcsv_and_7.m
importcsv_ios_7.m
input: wifiprocess/uid*.csv, location/uid*.csv, activity/uid*.csv output: gps_feature_and.mat, gps_feature_ios.mat

data fusion - upsampling

upsample_and_8.m
upsample_ios_8.m
input: gps_feature_and.mat, gps_feature_ios.mat
output: gps_feature_and_up.mat, gps_feature_ios_up.mat

feature generation

gpsExtracion_main_9.m
feature_extract.m
input: gps_feature_and_up.mat, gps_feature_ios_up.mat
output: dSet_and.mat, dSet_ios.mat

tool scripts, don't need to take care

dbscan.m
gridLegend.m
lldistkm.m