topic Re: tPOP & Microsoft Basic Authentication Retirement in Talend Studio

tPOP & Microsoft Basic Authentication Retirement

JBristow — Fri, 15 Nov 2024 22:51:49 GMT

We have a job that extracts emails from an account using the IMAP format - and then we consume those emails and perform some automated processing. We've been notified that the service accounts we use are accessing their mail accounts via Basic Authentication (User Name & Password) - and Microsoft will be retiring Basic Authentication in October of 2022.

Looking at the tPOP component, there are no security options - with the exception of "use SSL" - which we have checked. Thus my assumption is that the component uses Basic Authentication.

Are there any plans to address this - or anyone else who might have a similar issue to extract emails for processing and uses a different approach?

Thanks in advance for any direction/response.

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Mon, 06 Jun 2022 07:44:00 GMT

Hello,

We're aware of the Basic Auth "deprecation". We're planning to include Oauth 2.0 (first for tSendMail component others to follow) as soon as the following feature is available: https://www.microsoft.com/en-us/microsoft-365/roadmap?filters=&searchterms=70577

Unfortunately the options available today (as far as we understand)

Basic auth is disabled by default but can be enabled
Basic auth might stay with us even after october
Microsoft expects everyone to use Oauth2
Microsoft provides only a few auth flows:
- Graph API would grant access to all the mails and mailboxes within an organization.
- OAuth2 flows that require 2 step auth (or URL opening, etc) are very good for security but not a feasible option for daemon/ETL

Introducing the XOauth2.0 protocol effort is tracked by TDI-47369 In case you have access to support feel free to raise a case so you can be notified.

I'll leave a few links here on which I based this entry:

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-deprecation-in-exchange-online-may-2022/bc-p/3391016

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-deprecation-in-exchange-online-may-2022/bc-p/3391016/highlight/true#M33026

https://eclipse-ee4j.github.io/mail/OAuth2

https://github.com/eclipse-ee4j/mail/issues/461

https://docs.microsoft.com/en-us/azure/active-directory/develop/migrate-adal-msal-java

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-september-2021-update/ba-p/2772210

Re: tPOP & Microsoft Basic Authentication Retirement

JBristow — Fri, 10 Jun 2022 17:01:04 GMT

Thank You! We use tSendMail in a number of jobs, but as to an SMTP without authentication. Not sure if we'll be impacted there. We absolutely will on the tPOP as we perform an IMAP Poll of Inbox emails and pull those down for automated processing.

We're currently on 7.2.1 - upgrading to 8.0.1 in process. Is it a safe assumption that any "fix" to address this would not be retro-fitted to 7.2.1?

Thanks!

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Fri, 10 Jun 2022 17:48:36 GMT

Hello,

7.2 reaches its end of life this month. There are no patches planned for it after that. As for tSendMail / tPOP: The Oauth2 support will be added for Talend 8. As soon as there's a workflow we could rely on. We're almost in middle of June and the microsoft feature is still in Development. https://www.microsoft.com/en-us/microsoft-365/roadmap?filters=&searchterms=70577

I've created a ticket internally to make sure we'll analyze the other components where this might be required. Such as tPOP.

Re: tPOP & Microsoft Basic Authentication Retirement

DBLONDEL1643728674 — Fri, 26 Aug 2022 09:12:49 GMT

Bonjour,

Any news on this item ?

Microsoft announced to stop basic authentication by October 1st 2022 !!!

What are the alternative proposed by talend for tSendMail and tPop ?

Is there a cookbook somewhere ?

Thanks in advance for your help.

regards

Damien

Re: tPOP & Microsoft Basic Authentication Retirement

JBristow — Fri, 26 Aug 2022 14:36:52 GMT

Talend released the Monthly Studio Patch - R202208 - which has added an "Authentication Mode" option to the component. I just got this installed yesterday and am working on validation. Basically it looks as though if you're going to switch over to OAuth access - which we are doing - then you'll need to add logic to get your OAuth Access Token - and then in the tPOP you set the mode to "OAUTH2" and provide the User ID and Oauth Access Token instead of User ID and Password.

I'll be testing this out next week and will update.

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Mon, 29 Aug 2022 16:28:47 GMT

Hello Damien,

Microsoft doesn't offer a non-interactive flow for SMTP protocol. You can see it here: https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#smtp-protocol-exchange

For tSendMail / tPOP in the August release we've added OAuth Access token, but the token generation is not included.

We're working on adding the token generation to tPOP, (and backporting the Talend 8 Access token to 7.3) however it won't be for tSendMail as that requires a feature to be implemented by Microsoft itself.

Regards,

Balázs

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Mon, 29 Aug 2022 17:07:14 GMT

Hello @Johnie Bristow

Yes that is correct. I'm going to provide a step-by-step guide how to set up Applications / configure them and how to obtain such tokens via a routine. I have the steps, the java code ready just need to finalize the community post.

This is the Microsoft guide:

https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth

Re: tPOP & Microsoft Basic Authentication Retirement

DBLONDEL1643728674 — Mon, 05 Sep 2022 20:37:41 GMT

Bonjour @JBristow and @Balazs Gunics ,

Did you progress ?

On my side I am using MS Graph API to get a token ... based on parameter we get from Azure (defining an app : tenant, client ID, Client secret).

Then our system teams associated the email adress to this app ...

we get a token ...

Then, after patching, when we try to implement the tPop with the "OAuth2" option we have error reject "Protocol error. Connection is closed. 10"

Any idea ?

regards

Damien

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Tue, 06 Sep 2022 06:45:49 GMT

Hello,

Microsoft went on a different path than Google and their OAuth exchange is in 2 lines instead of 1 line. For this reason there's an extra checkbox under Advanced Settings that needs to be enabled: Use two line Authentication for OAUth2

Google Gmail doesn't require this, Microsoft Exchange does.

Re: tPOP & Microsoft Basic Authentication Retirement

JBristow — Wed, 14 Sep 2022 19:42:00 GMT

We are just now getting resources to start this. We also have issues with 8.0.1 and cannot go "live" at this time so we are having to re-design our 7.2.1 job to grab the Token and then access the mailbox in customized java code. We'll grab the token via an API call (Microsoft Graph API) with an application key and a secret key - and once we have the access token we'll extract the emails from the INBOX we need. Then when we finally get 8.0.1 working and can go "live" with it - we'll convert the job to use the new tPOP component.

Complicated process change as Basic Authentication was easy. lol.

I'll keep the tread updated once we have it working.

Thanks.

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Thu, 15 Sep 2022 14:50:12 GMT

Did you know that the component is open source so you can take a look how it will look with 8?

https://github.com/Talend/tdi-studio-se/tree/master/main/plugins/org.talend.designer.components.localprovider/components/tPOP

Re: tPOP & Microsoft Basic Authentication Retirement

DOSI_ALKOR — Fri, 16 Sep 2022 07:30:39 GMT

Will these changes be in R2022-09?

If so, do you have a release date ?

Thanks.

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Mon, 19 Sep 2022 10:48:42 GMT

Hello,

Yes these will be included in both 7.3 and 8 releases of September. 7.3 is already released and the component help page is reflecting the new parameters:

https://help.talend.com/r/en-US/7.3/pop/tpop-standard-properties

And the step by step guide for configuring the AD client is also available:

https://help.talend.com/r/en-US/7.3/pop/configuring-an-oauth2-application-for-pop-and-imap

Cloud customers can already download the 7.3 patch from the Downlaod sections of TMC, and the Talend 8 patch will be available in the update page soon: https://update.talend.com/Studio/8/updates/

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Mon, 19 Sep 2022 11:07:08 GMT

UPDATE 2023-02-17: Feature request for tSendMail + Exchange Auth: https://feedbackportal.microsoft.com/feedback/idea/c343ff42-a6ae-ed11-a81b-000d3a0450e3

Hello,

I'd like to provide an official update to this question / thread.

Microsoft is going to deprecate basic Auth (announcement , feedback) in the beginning of this October. This is done as Basic Auth is considered insecure. Talend mail components only support Basic Authentication, and while some providers provide Application passwords this functionality (feedback/feature request) is not available for Microsoft.

tSendMail

Talend jobs and components should be considered a daemon/service like application. For this it’s essential that there’s a non-interactive option for authentication. Microsoft recently made available Client Credential Flow support (announcement) for POP/IMAP. With this functionality it become possible to read e-mails. However in order to send e-mails one would traditionally rely on the SMTP protocol. As of the middle of September 2022 there’s still no support for SMTP with Client Credential flow.

This means that starting from October Microsoft will seemingly disable Basic authentication without providing a proper secure solution that can be used from daemon/service like applications. (No Non-interactive flow for SMTP , Confirmation from Microsoft Exchange team member )

The tSendMail component uses SMTP protocol and won't be affected by this change of Microsoft.

UPDATE 2022-10-06: To our current knowledge there's no non-interactive flow available for the SMTP protocol. This means that the same Microsoft Exchange auth type that is available for tPOP won't work with SendMail hence it wasn't added. In case there'll be a flow that can be used to generate tokens the token can be passed via the OAuth2 auth type as an Access Token. It might be necessary to enable 2 line auth under the Advanced settings.

It is also possible to add more dropdown options to make the token generation easier but these options need to support scheduled task executions where human interaction is not possible.

tPOP

The tPOP component uses POP / IMAP and will be impacted. Both of these components got their Authentication options modified and now have Oauth access token available next to the Basic Auth. If a token is presented the component can send/read e-mails. Such token can be generated via routines / external applications. This was introduced with Talend 8 R2022-08 and 7.3.1 R2022-09 releases. This should be a universal solution that can be used with any e-mail provider, and Oauth workflow.

Due to high demand we’ve also added support for Client Credential Flow in the tPOP component available as Microsoft Exchange auth. This will make the component to negotiate / retrieve an access token using the Microsoft Secure Authentication Library (MSAL). This was/going to be introduced Talend 7.3.1 R2022-09 and 8 R2022-09 releases. The necessary configuration steps can be found here: https://help.talend.com/r/en-US/8.0/pop/registering-microsoft-azure-application-for-pop-imap

Debug logs (UPDATE 2022-10-18)

Under the Advanced settings you can specify Custom properties. Adding the following entries will generate more logs about the debug steps. This will include the token value generated during the process.

"mail.debug" "true"

"mail.debug.auth" "true"

I hope this can be accepted as an answer to this question.

Regards,

Balázs

Re: tPOP & Microsoft Basic Authentication Retirement

NAndrade1645029267 — Mon, 17 Oct 2022 18:31:46 GMT

Has anyone been able to successfully set up the authentication mode for Microsoft Exchange or OAuth? We have a Talend case open but its been confusing and still isn't working. We were told to create an app registration in Microsoft Azure. Anyone using Microsoft Azure and can assist with your settings?

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Tue, 18 Oct 2022 10:18:25 GMT

Did you follow the official guide? https://help.talend.com/r/en-US/8.0/pop/registering-microsoft-azure-application-for-pop-imap

Re: tPOP & Microsoft Basic Authentication Retirement

NAndrade1645029267 — Tue, 18 Oct 2022 13:18:43 GMT

Yes I have followed these instructions. However when running the job in Talend we get an Authenticate failed error. Please see my attached image showing our setting in Talend. Does this look correct? I am still waiting for someone from Talend to assist.

Re: tPOP & Microsoft Basic Authentication Retirement

NAndrade1645029267 — Tue, 18 Oct 2022 13:20:04 GMT

Re: tPOP & Microsoft Basic Authentication Retirement

Anonymous — Tue, 18 Oct 2022 16:41:31 GMT

This setup looks correct. Did you also do the PowerShell steps? Without that you'll never be able to access the e-mails.