Kafka Connect specifies two configuration items for retrying errors and backoff:
errors.retry.timeout=-1 #number of retries (-1 for infinite retries)
errors.retry.delay.max.ms=X #defaults to 1 min
The documentation suggests this takes effect when an error is thrown during processing. What it does not say, is that these items have no bearing once your sink connector receives the collection of records from Kafka. These settings only affect the Connect framework processing of the records.
I came across this issue because the documentation also says if you throw a RetriableException in your put() call in your sink processor it will resend the batch of records for processing again. I threw the exception and set the max delay above but tests were still timing out.
- The settings above have no affect on your processing of records in your connector. The default "retry" time is the time between commits and this is 1 minute.
- You have access to a context object in your connector. You can set the timeout on it right before throwing the RetriableException. This is not documented anywhere outside of this javadoc. The only clue this was what you had to do was the S3 sink connector doing it.