The World Bank’s Malaysia Hub has a new brief on the effect of immigrant workers on productivity (excerpt):
While unskilled immigrant workers have relatively low formal human capital, theory suggests that they can still contribute to productivity improvements by helping to increase efficiency and upgrading the skills of the native labor force. Empirical studies indicate that positive productivity effects do occur. This body of evidence does not provide a compelling argument for the closing of national borders to unskilled foreigners on economic grounds.
TL;DR version: The available evidence doesn’t support any negative impact on productivity, with some countries showing a positive impact. Note that this doesn’t necessarily preclude the possibility that reducing unskilled foreign workers will increase productivity, but it does make it unlikely.