How can we keep indexing working when the master server goes down?
I was told from the training that an instance of Coveo search can have only one Master server with maximum 2 mirrors. However, I was asked that how we can keep indexing going if the master server goes down. Can we have master 2 servers in 2 different data center?
As I said in training, there is no support for more than one master CES server with Coveo for Sitecore on-premises. This is because a Sitecore index can be configured to push only to one CES server. Additionally, Coveo for Sitecore UI components always query the first Coveo index that matches the context database. So having more than one index per database is useless because if the first master server stops, the queries will never be sent to the other ones if you have many.
One "solution" could be to setup multiple Coveo indexes per database in the Sitecore configuration files. Example for the master database:
You could configure each of those indexes to push on a different Coveo master CES server using specific index configuration instead of the default one. With this setup, each CES master server would receive documents to index. Those CES master servers would be totally independant with no synchronization between them. As I said, the UI components would only query the first ones, never the other ones.
In case of a halt of the first CES master server, you could manually comment the first set of Sitecore indexes in your Coveo for Sitecore configuration files for all your CM/CD Sitecore servers. Then, the UI components will fallback on the second set of indexes who are located on a different set of CES master/mirror servers. This comes with a downtime as Sitecore needs to reload the configuration files from the hard disk. Any pending indexing operation would be stopped by doing so.
This solution also costs more in term of licencing and hosting.
I hope this helps,