Exchange 2013: Outlook Anywhere periodicaly disconnects and reconnects

For some reason when external use Outlook Anywhere, after a while it disconnects from the Exchange server and tries to reconnect, then in few minutes it successfully connects again. All this time all the Exchange servers stay untouched.

Many of our users are experiencing this issue.

Tested the scenario with different versions of outlook, 2010, 2013, 2016. all of them experience the disconnects. I have tested at various locations with wired ethernet connection and WiFi. At all the locations the issue persists. Exchange cached mode is off, it seems it does not happen when cached mode is on!

I haven't spotted any relevant events in the eventviewer on the Exchange and TMG servers.

Our environment consists of a number of exchange servers, we use MAPI over HTTP connections to connect to the mailboxes. Autodiscover is configured. We make use of a TMG firewall server when connecting from outside of the LAN. 

All of the Exchange servers run Windows Server 2012 R2, the TMG server runs on Windows Server 2008R2.

here are the various Exchange servers, and their roles:

ServerRole          : Mailbox, ClientAccess
Edition             : Enterprise
AdminDisplayVersion : Version 15.0 (Build 1076.9)

ServerRole          : Mailbox, ClientAccess
Edition             : Enterprise
AdminDisplayVersion : Version 15.0 (Build 1076.9)

ServerRole          : ClientAccess
Edition             : Enterprise
AdminDisplayVersion : Version 15.0 (Build 1076.9)

ServerRole          : Mailbox, ClientAccess
Edition             : Enterprise
AdminDisplayVersion : Version 15.0 (Build 1076.9)

this are the Exchange MAPI and Outlook Anywhere virtual directories and their URL's:

        MAPI:
        InternalUrl                    ExternalUrl
        -----------                    -----------
        https://mail.domain.com/mapi https://mail.domain.com/mapi
        https://mail.domain.com/mapi https://mail.domain.com/mapi
        https://mail.domain.com/mapi https://mail.domain.com/mapi
        https://mail.domain.com/mapi https://mail.domain.com/mapi


        Outlook Anywhere:

        ExternalHostname  InternalHostname
        ----------------  ----------------
        mail.domain.com casserver.domain.com
        mail.domain.com casserver.domain.com
        mail.domain.com casserver.domain.com

How can I resolve this issue?