Solved: tPOP & Microsoft Basic Authentication Retirement - Qlik Community

JBristow · ‎2022-06-02

We have a job that extracts emails from an account using the IMAP format - and then we consume those emails and perform some automated processing. We've been notified that the service accounts we use are accessing their mail accounts via Basic Authentication (User Name & Password) - and Microsoft will be retiring Basic Authentication in October of 2022.

Looking at the tPOP component, there are no security options - with the exception of "use SSL" - which we have checked. Thus my assumption is that the component uses Basic Authentication.

Are there any plans to address this - or anyone else who might have a similar issue to extract emails for processing and uses a different approach?

Thanks in advance for any direction/response.

Anonymous · ‎2022-09-19

UPDATE 2023-02-17: Feature request for tSendMail + Exchange Auth: https://feedbackportal.microsoft.com/feedback/idea/c343ff42-a6ae-ed11-a81b-000d3a0450e3

Hello,

I'd like to provide an official update to this question / thread.

Microsoft is going to deprecate basic Auth (announcement , feedback) in the beginning of this October. This is done as Basic Auth is considered insecure. Talend mail components only support Basic Authentication, and while some providers provide Application passwords this functionality (feedback/feature request) is not available for Microsoft.

tSendMail

Talend jobs and components should be considered a daemon/service like application. For this it’s essential that there’s a non-interactive option for authentication. Microsoft recently made available Client Credential Flow support (announcement) for POP/IMAP. With this functionality it become possible to read e-mails. However in order to send e-mails one would traditionally rely on the SMTP protocol. As of the middle of September 2022 there’s still no support for SMTP with Client Credential flow.

This means that starting from October Microsoft will seemingly disable Basic authentication without providing a proper secure solution that can be used from daemon/service like applications. (No Non-interactive flow for SMTP , Confirmation from Microsoft Exchange team member )

The tSendMail component uses SMTP protocol and won't be affected by this change of Microsoft.

UPDATE 2022-10-06: To our current knowledge there's no non-interactive flow available for the SMTP protocol. This means that the same Microsoft Exchange auth type that is available for tPOP won't work with SendMail hence it wasn't added. In case there'll be a flow that can be used to generate tokens the token can be passed via the OAuth2 auth type as an Access Token. It might be necessary to enable 2 line auth under the Advanced settings.

It is also possible to add more dropdown options to make the token generation easier but these options need to support scheduled task executions where human interaction is not possible.

tPOP

The tPOP component uses POP / IMAP and will be impacted. Both of these components got their Authentication options modified and now have Oauth access token available next to the Basic Auth. If a token is presented the component can send/read e-mails. Such token can be generated via routines / external applications. This was introduced with Talend 8 R2022-08 and 7.3.1 R2022-09 releases. This should be a universal solution that can be used with any e-mail provider, and Oauth workflow.

Due to high demand we’ve also added support for Client Credential Flow in the tPOP component available as Microsoft Exchange auth. This will make the component to negotiate / retrieve an access token using the Microsoft Secure Authentication Library (MSAL). This was/going to be introduced Talend 7.3.1 R2022-09 and 8 R2022-09 releases. The necessary configuration steps can be found here: https://help.talend.com/r/en-US/8.0/pop/registering-microsoft-azure-application-for-pop-imap

Debug logs (UPDATE 2022-10-18)

Under the Advanced settings you can specify Custom properties. Adding the following entries will generate more logs about the debug steps. This will include the token value generated during the process.

"mail.debug" "true"

"mail.debug.auth" "true"

I hope this can be accepted as an answer to this question.

Regards,

Balázs

View solution in original post

Anonymous · ‎2022-06-06

Hello,

We're aware of the Basic Auth "deprecation". We're planning to include Oauth 2.0 (first for tSendMail component others to follow) as soon as the following feature is available: https://www.microsoft.com/en-us/microsoft-365/roadmap?filters=&searchterms=70577

Unfortunately the options available today (as far as we understand)

Basic auth is disabled by default but can be enabled
Basic auth might stay with us even after october
Microsoft expects everyone to use Oauth2
Microsoft provides only a few auth flows:
- Graph API would grant access to all the mails and mailboxes within an organization.
- OAuth2 flows that require 2 step auth (or URL opening, etc) are very good for security but not a feasible option for daemon/ETL

Introducing the XOauth2.0 protocol effort is tracked by TDI-47369 In case you have access to support feel free to raise a case so you can be notified.

I'll leave a few links here on which I based this entry:

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-deprecation-in-exchange-online-may-2022/bc-p/3391016

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-deprecation-in-exchange-online-may-2022/bc-p/3391016/highlight/true#M33026

https://eclipse-ee4j.github.io/mail/OAuth2

https://github.com/eclipse-ee4j/mail/issues/461

https://docs.microsoft.com/en-us/azure/active-directory/develop/migrate-adal-msal-java

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-september-2021-update/ba-p/2772210

JBristow · ‎2022-06-10

Thank You! We use tSendMail in a number of jobs, but as to an SMTP without authentication. Not sure if we'll be impacted there. We absolutely will on the tPOP as we perform an IMAP Poll of Inbox emails and pull those down for automated processing.

We're currently on 7.2.1 - upgrading to 8.0.1 in process. Is it a safe assumption that any "fix" to address this would not be retro-fitted to 7.2.1?

Thanks!

Anonymous · ‎2022-06-10

Hello,

7.2 reaches its end of life this month. There are no patches planned for it after that. As for tSendMail / tPOP: The Oauth2 support will be added for Talend 8. As soon as there's a workflow we could rely on. We're almost in middle of June and the microsoft feature is still in Development. https://www.microsoft.com/en-us/microsoft-365/roadmap?filters=&searchterms=70577

I've created a ticket internally to make sure we'll analyze the other components where this might be required. Such as tPOP.

DBLONDEL1643728674 · ‎2022-08-26

Bonjour,

Any news on this item ?

Microsoft announced to stop basic authentication by October 1st 2022 !!!

What are the alternative proposed by talend for tSendMail and tPop ?

Is there a cookbook somewhere ?

Thanks in advance for your help.

regards

Damien

JBristow · ‎2022-08-26

Talend released the Monthly Studio Patch - R202208 - which has added an "Authentication Mode" option to the component. I just got this installed yesterday and am working on validation. Basically it looks as though if you're going to switch over to OAuth access - which we are doing - then you'll need to add logic to get your OAuth Access Token - and then in the tPOP you set the mode to "OAUTH2" and provide the User ID and Oauth Access Token instead of User ID and Password.

I'll be testing this out next week and will update.

Anonymous · ‎2022-08-29

Hello Damien,

Microsoft doesn't offer a non-interactive flow for SMTP protocol. You can see it here: https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#smtp-protocol-exchange

For tSendMail / tPOP in the August release we've added OAuth Access token, but the token generation is not included.

We're working on adding the token generation to tPOP, (and backporting the Talend 8 Access token to 7.3) however it won't be for tSendMail as that requires a feature to be implemented by Microsoft itself.

Regards,

Balázs

Anonymous · ‎2022-08-29

Hello @Johnie Bristow

Yes that is correct. I'm going to provide a step-by-step guide how to set up Applications / configure them and how to obtain such tokens via a routine. I have the steps, the java code ready just need to finalize the community post.

This is the Microsoft guide:

https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth

DBLONDEL1643728674 · ‎2022-09-05

Bonjour @JBristow and @Balazs Gunics ,

Did you progress ?

On my side I am using MS Graph API to get a token ... based on parameter we get from Azure (defining an app : tenant, client ID, Client secret).

Then our system teams associated the email adress to this app ...

we get a token ...

Then, after patching, when we try to implement the tPop with the "OAuth2" option we have error reject "Protocol error. Connection is closed. 10"

Any idea ?

regards

Damien

Anonymous · ‎2022-09-06

Hello,

Microsoft went on a different path than Google and their OAuth exchange is in 2 lines instead of 1 line. For this reason there's an extra checkbox under Advanced Settings that needs to be enabled: Use two line Authentication for OAUth2

Google Gmail doesn't require this, Microsoft Exchange does.

tPOP & Microsoft Basic Authentication Retirement

Big Data

v8.x