Problem with missing values for categorical variables.

[Yan77us]

The problem of identifying missing values in dataset. The sample as following

data = {
    'Age': [35, 42, 30, 29, 51, 38],  
    'Gender': [1, 0, 1, 0, 0, 1],  
    'Income': [45000, np.nan, 65000, 48000, 70000, np.nan],  
    'Education': ['2', '1', '1', '', '1', '2'],  
    'Satisfaction': [4.5, 3.2, np.nan, 4.8, 3.9, 4.1]  
}
data = pd.DataFrame(data)

cat_col = ['Gender', 'Education']
group = 'Gender'

data_table1 = tb.TableOne(data, categorical = cat_col, 
    groupby = group, pval= True, htest_name=True,  decimals=3)

print(data_table1)

The output for Education with missing values has a separate category. If I convert to str, then the 'None' will be a separate category.

Please advise how to deal with. Thank.

[epabarker]

I've been dealing with a similar situation. I'm not quite sure what you mean, but I can interpret a few things.

• If you want '' to be considered a missing value, much the same as the np.nan missing values:
  • data.replace('', np.nan, inplace=True) # replace empty strings with NaN
• If you left it at that, you would get the following output, with the "None" category as you mentioned earlier.

		Missing	Overall	0	1	P-Value	Test
n			6	3	3
Age, mean (SD)		0	37.500 (8.216)	40.667 (11.060)	34.333 (4.041)	0.432	Welch’s T-test
Gender, n (%)	0		3 (50.000)	3 (100.000)	0 (0.000)	0.100	Fisher's exact
	1		3 (50.000)	0 (0.000)	3 (100.000)
Income, mean (SD)		2	57000.000 (12355.835)	59000.000 (15556.349)	55000.000 (14142.136)	0.813	Welch’s T-test
Education, n (%)	1		3 (50.000)	2 (66.667)	1 (33.333)	0.189	Chi-squared (warning: expected count < 5)
	2		2 (33.333)	0 (0.000)	2 (66.667)
	None		1 (16.667)	1 (33.333)	0 (0.000)
Satisfaction, mean (SD)		1	4.100 (0.612)	3.967 (0.802)	4.300 (0.283)	0.562	Welch’s T-test

• By default, include_null = True, which means None/Null values will be included as a category level
• If you want to treat '' as a missing value, and have the number of missing values appear in the missing column, you need to both set '' to np.nan, and set include_null=False when creating the TableOne object.
• If I remember correctly, tableone only considers np.nan as a missing value
• If you do the above, then you will end up with this:

		Missing	Overall	0	1	P-Value	Test
n			6	3	3
Age, mean (SD)		0	37.500 (8.216)	40.667 (11.060)	34.333 (4.041)	0.432	Welch’s T-test
Gender, n (%)	0	0	3 (50.000)	3 (100.000)	0 (0.000)	0.100	Fisher's exact
	1		3 (50.000)	0 (0.000)	3 (100.000)
Income, mean (SD)		2	57000.000 (12355.835)	59000.000 (15556.349)	55000.000 (14142.136)	0.813	Welch’s T-test
Education, n (%)	1	1	3 (60.000)	2 (100.000)	1 (33.333)	0.400	Fisher's exact
	2		2 (40.000)	0 (0.000)	2 (66.667)
Satisfaction, mean (SD)		1	4.100 (0.612)	3.967 (0.802)	4.300 (0.283)	0.562	Welch’s T-test

[tompollard]

Thanks @epabarker! It sounds like handling of missing values etc should be improved. Please let me know if you have suggestions and we will try to implement them over the next few weeks.

[Yan77us]

Thanks @epabarker It works!