Hard to comment without looking at the error description.
However, below are the potential errors I see here.
Single quote missing at the end of the expression.
Assuming that TimeColumn is holing QV time values , i would recommend to convert the sting literal to proper QV time (using TIME#) so that Qlik engine use internal numeric values during the comparison.
Example-if(TimeColumn<Time#('16:00:00 PM','hh:mm:ssTT') AND TimeColumn>=Time#('1:00:00AM','hh:mm:ssTT'), 'DayShift','NightShift')