Redmine

the same goes for gmail. they disabled basic authentication in last May.

I have a same issue.

Hello All,

Same issue on our side, we are fetching email from our office365 account with IMAP.
Any help will be welcome.

Thanks

Same Here

Adding weight to this Query - Basic Auth is deprecated this weekend (ish - it's MS). Can re enable till end of year after that OP will have to be removed if not compliant.

Any idea as to whether this will be picked up or not?

I have hotfixed lib/redmine/imap.rb with a fix to use xoauth2 when provided with a longer than 30 chars password.

require 'mail_xoauth2'

       if imap_options[:password].length > 30
           imap.authenticate('XOAUTH2', imap_options[:username], imap_options[:password])
       else
           imap.login(imap_options[:username], imap_options[:password]) unless imap_options[:username].nil?
       end

And then use the OAuth2 gem, to fetch a token from microsoft in cron script, which call 'rake redmine:email:receive_imap' with the token as password.
It's just a quick fix, but it works for now.

Hi Jesper

Firstly Thanks - This really helps.

Am I able to just swap out the library for a the new version (if so where do I get it) or do I need to do a full redmine update?

Hi,

we just worked with RedmineUP to get OAuth with O365 working with the Helpdesk Plugin.
No advertisement, but we had the same issue as Howard and wanted this to be done before anything is cut off by MS.

The plugin uses MS Graph from what i can see btw, not imap with with oauth.

FYI

Upgraded Ruby to the latest version and installed the mail_xoauth gem https://rubygems.org/gems/mail_xoauth2

Built a quick script to get the Access token from MS and used the token as the password in rake task

All working successfully - Fantastic !

Howard & Jester, this is great thank you.

Howard or Jester, would you share the quick script you built? And how often you scheduled it in cron?

(I'm scrambling to patch a server I am unfamiliar with, could use a leg up).

Thanks in advance!

Hi Darren,

Update the lib/redmine/imap.rb code to the new script as per the above CR
Install the OAuth2 gem onto the server

Script:
eval MSTOKEN=$(curl -X POST -H 'Content-type: application/x-www-form-urlencoded' -d "client_id=CLIENTID&scope=https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/User.Read&grant_type=password&username=EMAILADDRESS&password=PASSWORD&client_secret=CLIENTSECRET https://login.microsoftonline.com/DOMAIN/oauth2/v2.0/token | jq '.access_token')

openproject run bundle exec rake redmine:email:receive_imap host='outlook.office365.com' username='EMAILADDRESS' password=$MSTOKEN port=993 ssl=true allow_override=type,project project=OPPROJECTNAME unknown_user=accept no_permission_check=1

CronJob
*/1 * * * 1-5 sudo /home/Kempston/opraketask.sh

we run the script every time we process an email -> project using the rake task (every minute or so) Technically you need to renew the token every hour by deafult

Hi Raja,

Deprecation of MS Basic Auth does not remove the use of IMAP, it just means the your IMPA will not be able to use basic auth going forward.

The method here allows you to add OAUTH to your IMAP to make it compliant.

It is simply a case of authenticating IMAP with and access token instead of a password.

https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth

Hope this helps

I am trying to follow Howard Brown instructions but not making any progress.

The instructions are not quite in details so they are hard to follow.

Is the script below is for this file? /home/Kempston/opraketask.sh
Script:
eval MSTOKEN=$(curl -X POST -H 'Content-type: application/x-www-form-urlencoded' -d "client_id=CLIENTID&scope=https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/User.Read&grant_type=password&username=EMAILADDRESS&password=PASSWORD&client_secret=CLIENTSECRET https://login.microsoftonline.com/DOMAIN/oauth2/v2.0/token | jq '.access_token')

openproject run bundle exec rake redmine:email:receive_imap host='outlook.office365.com' username='EMAILADDRESS' password=$MSTOKEN port=993 ssl=true allow_override=type,project project=OPPROJECTNAME unknown_user=accept no_permission_check=1

CronJob
*/1 * * * 1-5 sudo /home/Kempston/opraketask.sh

Running the script under cron gives this error under /var/log/cron

Oct 13 15:19:01 redmine-01 CROND^1478036: (root) CMDOUT (curl: no URL specified!)
Oct 13 15:19:01 redmine-01 CROND^1478036: (root) CMDOUT (curl: try 'curl --help' or 'curl --manual' for more information)
Oct 13 15:20:01 redmine-01 CROND^1478077: (root) CMD (sudo /opt/opraketask.sh)
Oct 13 15:20:01 redmine-01 CROND^1478076: (root) CMDOUT (curl: no URL specified!)
Oct 13 15:20:01 redmine-01 CROND^1478076: (root) CMDOUT (curl: try 'curl --help' or 'curl --manual' for more information)
Oct 13 15:21:01 redmine-01 CROND^1478116: (root) CMD (sudo /opt/opraketask.sh)

I tied running latest Redmine 5.0.3 running on Ubuntu 20.04 and also tried running under Redmine version 4.2.2 on Linux
I have installed the OAuth2 gem as well.

[root@redmine-01 redmine-4.2.2]# ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
[root@redmine-01 redmine-4.2.2]# rails -v
Rails 5.2.6
[root@redmine-01 redmine-4.2.2]# gem -v
3.1.4

[root@redmine-01 redmine-4.2.2]# bundler -v
Bundler version 2.2.25

[root@redmine-01 redmine-4.2.2]# passenger -v
Phusion Passenger(R) 6.0.10

I have register my Redmine application under Azure AD using this links https://blog.rebex.net/oauth2-office365-rebex-mail

To get the Application (client) ID and also Redmine client secret ID.

When I run the rake command I get the following error.

• Invoke redmine:email:receive_imap (first_time)
• Invoke environment (first_time)
• Execute environment
• Execute redmine:email:receive_imap
  rake aborted!
  LoadError: cannot load such file -- mail_xoauth2
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /var/www/redmine-4.2.2/lib/redmine/imap.rb:20:in `<top (required)>'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:378:in `block in require_or_load'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:37:in `block in load_interlock'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies/interlock.rb:13:in `loading'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:37:in `load_interlock'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:356:in `require_or_load'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:510:in `load_missing_constant'
  /usr/local/rvm/gems/ruby-2.7.2/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:195:in `const_missing'
  /var/www/redmine-4.2.2/lib/tasks/email.rake:121:in `block (4 levels) in <top (required)>'
  /var/www/redmine-4.2.2/app/models/mailer.rb:649:in `with_synched_deliveries'
  /var/www/redmine-4.2.2/lib/tasks/email.rake:120:in `block (3 levels) in <top (required)>'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
  /usr/local/rvm/gems/ruby-2.7.2/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
  /usr/local/rvm/gems/ruby-2.7.2/bin/rake:23:in `load'
  /usr/local/rvm/gems/ruby-2.7.2/bin/rake:23:in `<main>'
  /usr/local/rvm/gems/ruby-2.7.2/bin/ruby_executable_hooks:24:in `eval'
  /usr/local/rvm/gems/ruby-2.7.2/bin/ruby_executable_hooks:24:in `<main>'
  Tasks: TOP => redmine:email:receive_imap
  [root@redmine-01 redmine]#

I have placed this code as decribed above in my lib/redmine/imap.rb

require 'mail_xoauth2'

if imap_options[:password].length > 30
           imap.authenticate('XOAUTH2', imap_options[:username], imap_options[:password])
       else
           imap.login(imap_options[:username], imap_options[:password]) unless imap_options[:username].nil?
       end
~

There's a type with a quote missing after client_secret=CLIENTSECRET

eval MSTOKEN=$(curl -X POST -H 'Content-type: application/x-www-form-urlencoded' -d "client_id=CLIENTID&scope=https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/User.Read&grant_type=password&username=EMAILADDRESS&password=PASSWORD&client_secret=CLIENTSECRET" https://login.microsoftonline.com/DOMAIN/oauth2/v2.0/token | jq '.access_token')

I'm also trying to get this working with either my current Redmine 4.1.1 or a new Redmine 5.0.2 but can't get the damn thing to connect.
I ended up adding the gem in the Gemfile using the instruction from the mail_xoauth2 website:

 gem 'mail_xoauth2', '~> 1.0', '>= 1.0.2'

and rebuilding the Redmine project after implementing the CR, it does retrieve the token but I always get "NO LOGIN".

I read that maybe the token needed to be encoded in base 64 or even encoded including the email username: as the "legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth" microsoft documentation page.

Would it be the case here ?
If so, does anyone have a working script for a Ubuntu based system ?

Because I've tried :
- converting token to base 64
- including the token in a string with user as described in the documentation and converting the resulting string to base64

• How to connect to office 365 with IMAP protocol on behalf of a user keeping offline access
• How to connect to Office 365 with IMAP, Oauth2 and Client Credential Grant Flow

Since my company moved mailing from a self-hosted solution to O365, Forwarding emails from your email server has been broken.

In order to workaround the broken mail server getmail6 is used to listen for incoming mails (using IMAP IDLE). The notes below may be missing some details - feel free to ask if something is not clear.

Since this redmine blocks posts containing many external links (see #12495), I had to post my solution in several edits.

OAuth2¶

Getmail contains a helper to retrieve OAuth2 tokens: getmail-gmail-xoauth-tokens. As explained in Using Configured Getmail6 getmail-gmail-xoauth-tokens a JSON file is needed. In our case the file is ~/.getmail/microsoft.json.

The xxx-id-big-uuid is a placeholder for the Application ID. The Application ID was created as explained in -- How to create a Microsoft registration --.

Citing Using Configured Getmail6 getmail-gmail-xoauth-tokens

Start by running:

getmail-gmail-xoauth-tokens -i ~/.provider.json

It will print out a URL. Put that in your browser. Your browser will do 2FA (or not if you’re already known) and then redirect to http://localhost:10101/?code=xxx&session=yyy This redirect will fail, BUT you can cut-and-paste the URL and extract the xxx part (the code) and enter that in the getmail-gmail-xoauth-tokens program. After you do this you should see an access_token and a refresh_token in your (now rewriten) JSON file.

In order to complete the initialization a link has to be opened in a Javascript capable browser on the same machine getmail-gmail-xoauth-tokens --init .getmail/microsoft.json is called. The script listens on localhost:8083 for the reply. If the link is opened on another machine, e.g. your workstation, the redirect fails.

Instead of cut and paste as suggested above, the content of ~/.getmail has been copied to the local workstation (using rsync). Then the initialization has been done there and the result synced back to Redmine VM.

To test, call:

getmail-gmail-xoauth-tokens ~/.getmail/microsoft.json

getmail¶

The getmail files are in ~/.getmail. This is ~/.getmail/getmailrc:

[retriever]
type = SimpleIMAPSSLRetriever
server = outlook.office365.com
username = redmine@yourdomain.org
password_command = ("getmail-gmail-xoauth-tokens", "/home/youruser/.getmail/microsoft.json")
use_xoauth2 = true

[destination]
type = MDA_external
path = /home/youruser/redmine/rdm-mailhandler.rb
arguments = ("--url=https://redmine.yourdomain.org", "--key-file=/home/redmine/ws-api-key.txt", "--certificate-bundle=/home/redmine/certs/cacert.pem")
#type = Maildir
#path = ~/Maildir/

[options]
read_all = false

For an explanation see getmail documentation.

For testing the [destination] section had been configured to deliver to ~/Maildir. Then to test, call:

getmail

Or to listen for incoming mails:

getmail --idle=INBOX

Getmail had been started with:

getmail --idle=INBOX 2>&1 | logger &

rdm-mailhandler.rb¶

The script used for Forwarding emails from your email server has been downloaded from github (since our redmine runs in docker container):

wget https://raw.githubusercontent.com/redmine/redmine/master/extra/mail_handler/rdm-mailhandler.rb

To test, call:

cat ~/Maildir/new/1666039927.M876038P68753Q0R69e97bf81b6e27bf.redmine | ./rdm-mailhandler.rb --url=https://redmine.yourdomain.org --key-file=ws-api-key.txt --certificate-bundle=certs/cacert.pem

Where ~/Maildir/new/1666039927.M876038P68753Q0R69e97bf81b6e27bf.redmine is a mail you downloaded when testing getmail as explained above.

Cron¶

Using crontab -e the line below has been added to the crontab:

@reboot getmail --idle=INBOX

It should start getmail on the next reboot, but I did not test it yet.

Hello Lars M.

Thank you for this information,

I am trying to work on thia process, I got the first part working, this one.

root@redmine-03:~# getmail-gmail-xoauth-tokens --init ~/.getmail/microsoft.json
Visit this url to obtain a verification code:

It gives me the redmine application client ID from Microsoft Registration and Do have the Redmine Application Client ID, Cliient_Secret, Object ID and Directory(tenant) ID.

but the getmail part is NOT working for me, I have Redmine-5.0.3 running on Ubuntu 20.04 and our company is on Azure AD, Microsoft 365 admin center and Exchange on line as well.

This is my getmail configutaion file: getmailrc

root@redmine-03:~/.config/getmail# more getmailrc
[retriever]
type = SimpleIMAPSSLRetriever
server = outlook.office365.com
username = redmine@yourdomain.org
password_command = ("getmail-gmail-xoauth-tokens", "/root/.getmail/microsoft.json")
use_xoauth2 = true
arguments = ("--log-errors", )

[destination]
type = MDA_external
path = /var/www/redmine-5.0.3/extra/mail_handler/rdm-mailhandler.rb
arguments = ("--url=https://redmine.ipc.org", "--key-file=/home/redmine/ws-api-key.txt", "--certificate-bundle=/home/redmine/certs/gd_bundle-g2-g1.pem")
#type = Maildir
#path = ~/Maildir/

[options]
read_all = false
root@redmine-03:~/.config/getmail#

No matter how I run it, I get the same error: getmailrc: operation error (External program error (getmail-gmail-xoauth-tokens exited with 1))

root@redmine-03:~/.config/getmail# getmail --rcfile=getmailrc
getmail version 6.18.10
Copyright (C) 1998-2022 Charles Cazabon and others. Licensed under GNU GPL version 2.
SimpleIMAPSSLRetriever:redmine@yourdomain.org@outlook.office365.com:993:
getmailrc: operation error (External program error (getmail-gmail-xoauth-tokens exited with 1))
0 messages (0 bytes) retrieved, 0 skipped

root@redmine-03:~/.config/getmail# getmail
getmail version 6.18.10
Copyright (C) 1998-2022 Charles Cazabon and others. Licensed under GNU GPL version 2.
SimpleIMAPSSLRetriever:redmine@yourdomain.org@outlook.office365.com:993:
getmailrc: operation error (External program error (getmail-gmail-xoauth-tokens exited with 1))
0 messages (0 bytes) retrieved, 0 skipped

root@redmine-03:~/.config/getmail# getmail --idle=INBOX
getmail version 6.18.10
Copyright (C) 1998-2022 Charles Cazabon and others. Licensed under GNU GPL version 2.
SimpleIMAPSSLRetriever:redmine@yourdomain.org@outlook.office365.com:993:
getmailrc: operation error (External program error (getmail-gmail-xoauth-tokens exited with 1))
0 messages (0 bytes) retrieved, 0 skipped from SimpleIMAPSSLRetriever:redmine@yourdomain.org@outlook.office365.com:993
--idle given, but server does not support IDLE

Please advise,

Thank you for your time,

Arif

mis org wrote:

No matter how I run it, I get the same error: getmailrc: operation error (External program error (getmail-gmail-xoauth-tokens exited with 1))

Does getmail-gmail-xoauth-tokens work, if you call it directly? Like this:

getmail-gmail-xoauth-tokens /root/.getmail/microsoft.json

It should print the OAuth token on stdout and return 0. You can check the return value like this (immediately after running getmail-gmail-xoauth-tokens):

echo $?

If it fails, does it print any error message?

mis org wrote:

This is my getmail configutaion file: getmailrc

root@redmine-03:~/.config/getmail# more getmailrc
[retriever]
...
username = redmine@ yourdomain.org

Is this a copy/paste error?

Hello Lars M.

Thank you catching my mistake yourdomain.org in the getmailrc file, I have just fixed it, and when I run this command I get this error message.

root@redmine-03:~/.config/getmail# ls ltr /root/.getmail/microsoft.json
-rw-r--r- 1 root root 540 Oct 18 22:40 /root/.getmail/microsoft.json

root@redmine-03:~/.config/getmail# getmail-gmail-xoauth-tokens /root/.getmail/microsoft.json
Traceback (most recent call last):
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 153, in <module>
print("%s" % auth.token())
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 124, in token
if time.time() >= self.data.get('expires_at'):
TypeError: '>=' not supported between instances of 'float' and 'NoneType'
root@redmine-03:~/.config/getmail#

This is my python version on ubutu 20.04 redmine-02 Linux Server.

root@redmine-03:~/.config/getmail# python3 --version
Python 3.8.10

root@redmine-03:~/.config/getmail# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal
root@redmine-03:~/.config/getmail#

mis org wrote:

root@redmine-03:~/.config/getmail# getmail-gmail-xoauth-tokens /root/.getmail/microsoft.json
Traceback (most recent call last):
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 153, in <module>
print("%s" % auth.token())
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 124, in token
if time.time() >= self.data.get('expires_at'):
TypeError: '>=' not supported between instances of 'float' and 'NoneType'

To me it looks like your microsoft.json is not valid. The error indicates that self.data.get('expires_at') evaluates to 'NoneType'. I guess that your microsoft.json does not contain the field expires_at. After getmail-gmail-xoauth-tokens --init ~/.getmail/microsoft.json the microsoft.json contains a line like this (I replaced sensitive information by <...>):

{"scope": "<m$url>/IMAP.AccessAsUser.All <m$url>/POP.AccessAsUser.All <m$url>/SMTP.Send offline_access", "user": "mail@yourdomain.org", "client_id": "<your client id>", "client_secret": "", "token_uri": "m$url/oauth2/v2.0/token", "auth_uri": "<m$url>/oauth2/v2.0/authorize", "redirect_uri": "http://localhost", "access_token": "<very long token string>", "expires_at": 1666308614.85213, "refresh_token": "0.ATkA2_egoy_<very long token string>"}

Please have a look into your microsoft.json. It should contain only one line looking similar to the line above. If it does not, then the initialization was not successful or you are not passing the initialized microsoft.json to getmail-gmail-xoauth-tokens.

Thank you so much for your assistance.

I created the microsoft.json file from the link you provided earlier.

This is your note from above with the link:

Getmail contains a helper to retrieve OAuth2 tokens: getmail-gmail-xoauth-tokens. As explained in Using Configured Getmail6 getmail-gmail-xoauth-tokens a JSON file is needed. In our case the file is ~/.getmail/microsoft.json.

[[https://www3.isi.edu/~johnh/OTHER/LINUX/OAUTH2/index.html]]

This is the microsoft.json file in the above link with <m$url>/*common*/auth2/v2.0/authorize and <m$url>/*common*/oauth2/v2.0/token for both auth_uri & token_uri

For Microsoft (O365), it will be:

{
"user": "xxx-you@example.com",
"scope": "offline_access <m$url>/IMAP.AccessAsUser.All <m$url>/POP.AccessAsUser.All <m$url>m/SMTP.Send",
"client_id": "xxx-id-big-uuid",
"client_secret": "xxx-secret-big-charstring",
"tenant": "common",
"auth_uri": "<m$url>/common/oauth2/v2.0/authorize",
"token_uri": "<m$url>/common/oauth2/v2.0/token",
"redirect_uri": "http://localhost:10101"
}

And This below is my microsoft.json file.

root@redmine-03:~/.getmail# more microsoft.json {
"user": "redmine@ipc.org",
"scope": "offline_access <m$url>/IMAP.AccessAsUser.All <m$url>/POP.AccessAsUser.All <m$url>/SMTP.Send",
"client_id": "b4xxxxxxx-xxxxxd99-exxxx4-45xx-8xx7-dxxxxxe113",
"client_secret": "UpZxx~u4xx2Ch.6-xxxxxd2h9LpNnkljtco4iaSi",
"tenant": "common",
"auth_uri": "<m$url>/common/oauth2/v2.0/authorize",
"token_uri": "<m$url>/common/oauth2/v2.0/token",
"redirect_uri": "http://localhost:10101"
}

Also if I use your line above for microsoft.json file, then where do I get this value for <very long token string> ?

This is the value I got from Browser URL: is everything after code= and all the rest is my access_token?

http://localhost:8083/?code=0.AQ4ABr_Y2c9Ntku4sJ_bHAQOzJltQ7Qk7IBFh0fWOa7X4RMOAJs.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P-BbnfKm5zIijVTHpluOP87oObyKpUbt1-paJVTL7Uq6vIHfEUwkUeq4FHBur2dqrfxhGo3RGfnsn-7-Qn6upwwWGK1r6_Qvghp2bLFMVNGZK781sWxhayMiL17ELOeTZuTH9daWHGoNUJqhfOhapXE1cr8GJjbjYz7oaJzgIegczbOOFG57lsDoI-DsAGNo6-52E7ZsS0YD4gLNG-oKweQy0IcdJ_mLaz4tcjajMdFUpRli0iCDMWo-7m6CTHzbHw42SUaKc1vkzxqu8ER1T8godSsH3pVRJ6oZJbPZNe8mzDktt5K9J12Ee-T4yBftiJ82mQAMARew4uOb7ierXbfoPHsm4dfW6UqhWRI6m9O9j9N6jioKF5oWncexQD68zahvJXtTj85nCJgrG3pSTRjT3wb9Ui0RZt-RbcBul9K-xamMQ5BYz-Wo7RHdwco21zwIMyuqPfBfKXjT_uITBiDrXuhYZB_OaCRu-DYTwZZ7bFYirWjjMvr5J3an5NogrOjwsKOvOvweuFf9opGuoERIzywmRx6Elyi9_PC3mfNVZZneiTLropPrNakT8GCxowUQzy6-ZX3tL80YuRf0HV8gj3dgIzPLh0m2loyVqCmU4AEZM7LqJWpc4_h6vGdFVi-EfAg4UF5u_OGtf2hYveTHGLvkc1zvEXrZjT9tnWNIRTBamE0ecgmiN2uT_drotbDkilcwtpaLB2_ho7iQwrqO_YvjEKPOSNzbwo&session_state=eb92c9b1-8dbc-4eef-be94-399f6c3f8d5f

Thank you again,

mis org wrote:

And This below is my microsoft.json file.

root@redmine-03:~/.getmail# more microsoft.json {
"user": "redmine@ipc.org",
"scope": "offline_access <m$url>/IMAP.AccessAsUser.All <m$url>/POP.AccessAsUser.All <m$url>/SMTP.Send",
"client_id": "b4xxxxxxx-xxxxxd99-exxxx4-45xx-8xx7-dxxxxxe113",
"client_secret": "UpZxx~u4xx2Ch.6-xxxxxd2h9LpNnkljtco4iaSi",
"tenant": "common",
"auth_uri": "<m$url>/common/oauth2/v2.0/authorize",
"token_uri": "<m$url>/common/oauth2/v2.0/token",
"redirect_uri": "http://localhost:10101"
}

Why "redirect_uri": "http://localhost:10101"? It should be: "redirect_uri": "http://localhost:8083".

Also if I use your line above for microsoft.json file, then where do I get this value for <very long token string> ?

This microsoft.json file is the result of a proper getmail-gmail-xoauth-tokens --init .... Let me explain:

1. you call getmail-gmail-xoauth-tokens with the --init parameter and the initial microsoft.json file and you get a link to login.microsoftonline.com
2. you open that link and put in your m$ credentials
3. you are asked to grant getmail the permissions you specified before (as explained on -- How to create a Microsoft registration --)
4. login.microsoftonline.com will redirect you to http://localhost:8083 where getmail-gmail-xoauth-tokens is listening for m$' answer
5. getmail-gmail-xoauth-tokens will convert the microsoft.json to a one-liner containing all tokens and expiration dates needed. It then looks like I explained in my last comment.
6. on subsequent calls of getmail-gmail-xoauth-tokens you pass this converted microsoft.json as parameter

This is the value I got from Browser URL: is everything after code= and all the rest is my access_token?

http://localhost:8083/?code=0.AQ4ABr_Y2c9Ntku4sJ_bHAQOzJltQ7Qk7IBFh0fWOa7X4RMOAJs.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P-BbnfKm5zIijVTHpluOP87oObyKpUbt1-paJVTL7Uq6vIHfEUwkUeq4FHBur2dqrfxhGo3RGfnsn-7-Qn6upwwWGK1r6_Qvghp2bLFMVNGZK781sWxhayMiL17ELOeTZuTH9daWHGoNUJqhfOhapXE1cr8GJjbjYz7oaJzgIegczbOOFG57lsDoI-DsAGNo6-52E7ZsS0YD4gLNG-oKweQy0IcdJ_mLaz4tcjajMdFUpRli0iCDMWo-7m6CTHzbHw42SUaKc1vkzxqu8ER1T8godSsH3pVRJ6oZJbPZNe8mzDktt5K9J12Ee-T4yBftiJ82mQAMARew4uOb7ierXbfoPHsm4dfW6UqhWRI6m9O9j9N6jioKF5oWncexQD68zahvJXtTj85nCJgrG3pSTRjT3wb9Ui0RZt-RbcBul9K-xamMQ5BYz-Wo7RHdwco21zwIMyuqPfBfKXjT_uITBiDrXuhYZB_OaCRu-DYTwZZ7bFYirWjjMvr5J3an5NogrOjwsKOvOvweuFf9opGuoERIzywmRx6Elyi9_PC3mfNVZZneiTLropPrNakT8GCxowUQzy6-ZX3tL80YuRf0HV8gj3dgIzPLh0m2loyVqCmU4AEZM7LqJWpc4_h6vGdFVi-EfAg4UF5u_OGtf2hYveTHGLvkc1zvEXrZjT9tnWNIRTBamE0ecgmiN2uT_drotbDkilcwtpaLB2_ho7iQwrqO_YvjEKPOSNzbwo&session_state=eb92c9b1-8dbc-4eef-be94-399f6c3f8d5f

This is the redirect link created by login.microsoftonline.com. As you can see the port is 8083 and not 10101. So please fix your initial microsoft.json and repeat:

getmail-gmail-xoauth-tokens --init /root/.getmail/microsoft.json

By the way. There is no need to do this as root user. Using root for tasks like this is strongly discouraged.

Nicolas Metters wrote:

There's a type with a quote missing after client_secret=CLIENTSECRET
[...]

What DOMAIN field should be replaced by exactly ?

I Receive no TOKEN from the CURL command so what could be possibly wrong ?

curl -X POST -H 'Content-type: application/x-www-form-urlencoded' -d "client_id=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&scope=https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/User.Read&grant_type=password&username=XXXXXXXXXX@XXXXXXXXXXX.XXX&password=XXXXXXXXXX&client_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" https://login.microsoftonline.com/DOMAIN/oauth2/v2.0/token | jq '.access_token'

The result of the command is : NULL ...

What should I put instead of DOMAIN too ? is it the Tenant ID ?

Hello Lars M.

We are having an issue that Microsoft is not accepting http for the redirect URI, and we do not know how to get the
getmail-gmail-xoauth-tokens --init microsoft.json to listen on https.

We get this error on the redirect:

Sorry, but we’re having trouble signing you in.
AADSTS500117: The reply uri specified in the request isn't using a secure scheme.

Please advise?
Thanks for your time,

mis org wrote:

Sorry, but we’re having trouble signing you in.
AADSTS500117: The reply uri specified in the request isn't using a secure scheme.

Can you please post the reply URI?

Lars M.

It should be noted that getmail-gmail-xoauth-tokens only has localhost in it's verification URL , so using this exact url would not allow us to completed the verification because it was not running on our local machine.
So we were modifying the URL to point to our server name instead of localhost.

Here is the reply URL below.

http://redmine.ipc.org:8083

mis org wrote:

It should be noted that getmail-gmail-xoauth-tokens only has localhost in it's verification URL , so using this exact url would not allow us to completed the verification because it was not running on our local machine.
So we were modifying the URL to point to our server name instead of localhost.

Here is the reply URL below.

http://redmine.ipc.org:8083

Um, yes. This does not work. As explained in my initial post. Here again, in more detail:

In order to complete the initialization the link login.microsoftonline.com has to be opened in a Javascript capable browser on the same machine getmail-gmail-xoauth-tokens --init .getmail/microsoft.json is called. The script listens on localhost:8083 for the reply. If the link is opened on another machine, e.g. your workstation, the redirect fails.

If your Redmine is running on a headless server and you cannot open the link to in a Javascript capable browser there are two ways to work around this issue:

A) Citing Using Configured Getmail6 getmail-gmail-xoauth-tokens

Start by running:

getmail-gmail-xoauth-tokens -i ~/.provider.json

It will print out a URL. Put that in your browser. Your browser will do 2FA (or not if you’re already known) and then redirect to http://localhost:10101/?code=xxx&session=yyy This redirect will fail, BUT you can cut-and-paste the URL and extract the xxx part (the code) and enter that in the getmail-gmail-xoauth-tokens program. After you do this you should see an access_token and a refresh_token in your (now rewriten) JSON file.

B) Instead of cut and paste as suggested in A), the content of ~/.getmail can been copied to the local workstation (e.g. using rsync). Then the initialization can been done on your local workstation and the result has to be synced back to the Redmine server (the complete ~/.getmail directory or at least the converted microsoft.json file).

I recommend to use workaround B because then it is less likely to have copy/paste errors. I did not try workaround A myself.

Lars M.

Thank you for your help,

I am making progress, I had to install lite desktop for ubuntu 20.04 LTE on my Redmine server which comes with Firefox browser which is java based.
So now when I got the code page in the localhost FireFox browser.
I got this message returned in the local browser too:
"Your json file is updated."

Now which section Do I have to place the code-xxx part in my getmail-gmail-xoauth-tokens file OR I do not have to anymore.

and what are the next steps to test "Issue creation via Email" for Redmine ?

Thank you Again for your Help.

:-)

mis org wrote:

I got this message returned in the local browser too:
"Your json file is updated."

This is good.

Now which section Do I have to place the code-xxx part in my getmail-gmail-xoauth-tokens file OR I do not have to anymore.

You do not have to do it. Have look into your microsoft.json. You should find the tokens and expires_at fields as explained in https://www.redmine.org/issues/37688#note-23.

and what are the next steps to test "Issue creation via Email" for Redmine ?

Check if getmail-gmail-xoauth-tokens is working as explained in https://www.redmine.org/issues/37688#note-20

If it works, proceed with getmail as explained here: https://www.redmine.org/issues/37688#getmail

On a side note : because of a combination of factors, my company ended up purchasing a redmine Plugin that does manage OAuth token.

RedminUP's Helpdesk plugin, which has quite a good documentation:
https://www.redmineup.com/pages/help/helpdesk/setting-up-oauth-microsoft-outlook

But it has limitation so I'll still be trying to implement an alternative solution

Hello Lars M.

My local browser does display "Your josn file is updated." buy my microsoft.json file never gets updated.
I do have "expies_at" string in the microsoft.json file.

Here is my microsoft.json file.

{"scope": "<m$url>/IMAP.AccessAsUser.All <m$url>/POP.AccessAsUser.All <m$url>/SMTP.Send offline_access", "user": "redmine@ipc.org", "client_id": "b4436d99-ec24-4580-8747-d63xxxx7e113", "client_secret": "UpZ8Q~u4xxxxx.6-SKm5rd2h9LpNnkljtco4iaSi", "token_uri": "<m$url>/common/oauth2/v2.0/token", "auth_uri": "<m$url>/common/oauth2/v2.0/authorize", "redirect_uri": "http://localhost:8083", "expires_at": 1728692358.00230}

Not sure what is wrong now, Once I put the whole string after this command getmail-gmail-xoauth-tokens -i ~/.provider.json in the server local browser, it does ask me to sign in with the redmine user and accept the permission, then it display in the server local browser, this message "Your josn file is updated."

but my microsoft.json file never gets update with token information ?

By the way under the Azure AD redmine registration application, my Client Secrets expires on 10/12/2024 and use this link https://www.epochconverter.com/ to get my Epoch timestamp,

Epoch timestamp: 1728712061
Timestamp in milliseconds: 1728712061000

Description
Expires
Value
Secret ID
Redmine Client Secret
10/12/2024

This is message on the command line I get on the server.

redmine@redmine-03:~/.getmail$ getmail-gmail-xoauth-tokens --init microsoft5.json
Visit this url to obtain a verification code:
<m$url>/common/oauth2/v2.0/authorize?access_type=offline&client_id=b4436d99-ec24-4580-8747-d639aed7e113&prompt=consent&redirect_uri=http%3A%2F%2Flocalhost%3A8083%2F&response_type=code&scope=https%3A%2F%2Foutlook.office.com%2FIMAP.AccessAsUser.All%20https%3A%2F%2Foutlook.office.com%2FPOP.AccessAsUser.All%20https%3A%2F%2Foutlook.office.com%2FSMTP.Send%20offline_access

127.0.0.1 - - [22/Oct/2022 18:02:53] "GET /?code=0.AQ4ABr_Y2c9Ntku4sJ_bHAQOzJltQ7Qk7IBFh0fWOa7X4RMOAAA.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P95XgqAHUkQgeTqis5JHF0erbT_SPyD-ruRBpftwsvtkCvCCQJtI4TX4gK15Q7AV6RTbFL8nhM2WQUnTUqjQuFVSGoLyp0CbJkyUdITBN4u4mRgK_skT8Vs4GeWekuIX2M_tCLaa_hpWL_Bkh_ogiTzMH16oDaixa8Lka7B_U_pAFJ0_NM6wajmQouxzyHVU3t6q38kU2GjF7c0mE7AEh3SKNHX5VIjbyiUjd43SmyDuYWbGeafH6T4TdqnNxeRwKkpT-ttM1E5VKTqRUcCJN45ukTHhET5_f2qS8DAaV1Zzo464rNI67JEZA0iv0nRFv9IRwzpvcKJL-x7-QXICnqAG_c7ERRVMxEAD0miLeyYwEs87ZzjRGITglkk8RefaKXPhvmoiY7hJqnPVkusCkXSxboi7KsNkMV5SJtRU2uQcr5FEcqSNBcF6wvZWcHYvpzZNJFrSb-mnKK1B3kHzB5PLpShMXgFFTLxI3mNfpoQbMsILQ71cjRRUa8xC-RQXD_dE6E4mFWDQ-xzAVy4bZKlheRZlyXIrGHynuBFmQif1tbUecIx13NrKzkgxWwfROG5B1xC8Cq9We3mlrKt3LssYqQuFv8efryh9YUD9xQJsyfBuN_O2msVI-UOfTZa1-GQ3yRzLMtu6UMfe47A5Acf8hNJRdpMz2XZdqLQ8pnrABeamRKMt-0foFOCURRi9UIMsr7z0LIzSPK0ehQ4M4SdALTucJsWK3wDpg&session_state=dc2a43cd-8a4a-4653-9dad-cc9888f4ac91 HTTP/1.1" 200 -
Traceback (most recent call last):
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 148, in <module>
auth.init_tokens(oauthd.oauth_code,args.port)
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 113, in init_tokens
d = self.get_response(self.data['token_uri'], params)
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 93, in get_response
response = urllibrequest.urlopen(url, encoded).read()
File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
redmine@redmine-03:~/.getmail$

Please advise?

Thank you for your time,

Arif

mis org wrote:

Not sure what is wrong now, Once I put the whole string after this command getmail-gmail-xoauth-tokens -i ~/.provider.json in the server local browser, it does ask me to sign in with the redmine user and accept the permission, then it display in the server local browser, this message "Your josn file is updated."

This is the expected behavior.

This is message on the command line I get on the server.

redmine@redmine-03:~/.getmail$ getmail-gmail-xoauth-tokens --init microsoft5.json

Above you referred to ~/.provider.json, here you use microsoft5.json. Are you sure you are using the right json file when testing?

urllib.error.HTTPError: HTTP Error 401: Unauthorized

I found this bug report concerning Error 401 with getmail-gmail-xoauth-tokens: https://github.com/getmail6/getmail6/blob/master/docs/getmailrc-examples#L351

Unfortunately there is not one final solution.

#issuecomment-1197663127 proposes that "client_secret" in the JSON file may be wrong. I do not have a client secret in my file, so maybe that is why I did not hit this issue.
#issuecomment-1240561331 propose to remove "prompt=consent" from the URL. You could try this. But for me the URL worked with prompt=consent.
#issuecomment-1252243288 reports that he had this issue because of a mix of Python 3 and Pyhton 2 stuff. Maybe you should check your python setup as well.

Maybe you should re-check if your Azure settings conform to those explained here: https://github.com/getmail6/getmail6/blob/master/docs/getmailrc-examples

I found another bug report https://github.com/getmail6/getmail6/issues/135

...(I was getting “AADSTS700025: Client is public so neither 'client_assertion' nor 'client_secret' should be presented.").
There should probably also be a mention that the app's redir should be added as web app, I had it added as "mobile and desktop application" ...

I am no expert on OAuth2 but maybe in case you do have a client_secret for getmail, you have to register it as a web app, instead of mobile and desktop application (which is what I did and how it is explained for the mutt e-mail client).

Lars M.

If I do not use client_secret, then I get this message.

Traceback (most recent call last):
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 148, in <module>
auth.init_tokens(oauthd.oauth_code,args.port)
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 108, in init_tokens
params = self.copy('user', 'client_id', 'client_secret')
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 73, in copy
data[k] = self.data[k]
KeyError: 'client_secret'
redmine@redmine-03:~/.getmail$

Lars M.

I think I finally got it working, I ended up registering a brand new application registration under Azure AD using this link below, so something was wrong with the old application that I think I registered under Azure AD.

https://github.com/getmail6/getmail6/blob/master/docs/getmailrc-examples

I was able to place the message from this command in to local browser, and it did pasted all three sections(access_token, expires_at, and refresh_token) in the new microsoft6.json file

getmail-gmail-xoauth-tokens --init microsoft6.json

So the I was able to run this command and it did received 11 messages from redmine@ipc.org office 365 mailbox.
but with this error.
and I noticed it is appending the server name from the [retriever] section of getmailrc file. so username it is showing is as
redmine@ipc.org@outlook.office365.com:993:

Delivery error (command rdm-mailhandler.rb 23822 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))

redmine@redmine-03:~/.getmail$ getmail -r /home/redmine/.getmail/getmailrc
getmail version 6.18.10
Copyright (C) 1998-2022 Charles Cazabon and others. Licensed under GNU GPL version 2.
SimpleIMAPSSLRetriever:redmine@ipc.org@outlook.office365.com:993:
Delivery error (command rdm-mailhandler.rb 23822 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
msg 1/11 (39749 bytes), delivery error (command rdm-mailhandler.rb 23822 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
Delivery error (command rdm-mailhandler.rb 23824 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
msg 2/11 (60185 bytes), delivery error (command rdm-mailhandler.rb 23824 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
Delivery error (command rdm-mailhandler.rb 23826 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
msg 3/11 (162223 bytes), delivery error (command rdm-mailhandler.rb 23826 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
Delivery error (command rdm-mailhandler.rb 23828 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))

Thank you so much for all the help you have provided.

Thanks again,

-Arif

mis org wrote:

If I do not use client_secret, then I get this message.
...
KeyError: 'client_secret'

Just for the record: When I wrote I do not have a client secret in my file, I meant I set it to the empty string, i.e. "client_secret": "", and I do not have a client secret set in azure either.

mis org wrote:

Delivery error (command rdm-mailhandler.rb 23828 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))

These error messages are from rdm-mailhandler.rb. Verify that your email content and sender are valid.

Lars M.

Will check it out.

Also when I test sending messages to local MAILDIR, or any folder under /home/redmine/ user folder, it give me this message.
(path: maildir subdirectory "/home/redmine/Maildir/*cur*" does not exist)

Not sure where it is adding /cur at the end of any folder path and tells me it is incorrect path?

redmine@redmine-03:~/.getmail$ getmail
Configuration error: configuration file /home/redmine/.getmail/getmailrc incorrect (path: maildir subdirectory "/home/redmine/Maildir/cur" does not exist)
redmine@redmine-03:~/.getmail$ getmail --idle=INBOX
Configuration error: configuration file /home/redmine/.getmail/getmailrc incorrect (path: maildir subdirectory "/home/redmine/Maildir/cur" does not exist)

mis org wrote:

Also when I test sending messages to local MAILDIR, or any folder under /home/redmine/ user folder, it give me this message.
(path: maildir subdirectory "/home/redmine/Maildir/*cur*" does not exist)

Not sure where it is adding /cur at the end of any folder path and tells me it is incorrect path?

I did not mention it in my post, but it is documented in the getmail documentation:

The Maildir destination delivers to a qmail-style maildir. The maildir must already exist, and must contain all of the subdirectories required by the maildir format. getmail will not create the maildir if it does not exist. If you're not familiar with the maildir format, the requirements in a nutshell are: it must be a directory containing three writable subdirectories cur, new, and tmp, and they must all reside on the same filesystem.

Lars M.

Thank you so much for explaining it,

Will test it out,

Thanks again,

-Arif

Lars. M.

What If I want to use different username, like helpdesks@ipc.org in different microsoft1.json file.

How can re-generate or re-initialize the (OAuth 2.0) token under different username in microsoft1.json (different .json file for different user)

getmail-gmail-xoauth-tokens --init microsoft1.json

This new username have proper access to redmine.

Thank you,

-Arif

Lars M.

When I try to initilize different .json file microsoft1.json with different username, I get this.

redmine@redmine-03:~/.getmail$ getmail-gmail-xoauth-tokens --init microsoft1.json
Traceback (most recent call last):
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 140, in <module>
auth = OAuth2(args.tokenfile)
File "/usr/local/bin/getmail-gmail-xoauth-tokens", line 68, in init
self.data = json.load(f)
File "/usr/lib/python3.8/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 9 column 1 (char 597)
redmine@redmine-03:~/.getmail$

Nicolas Metters wrote:

There's a type with a quote missing after client_secret=CLIENTSECRET
[...]

Hello,

I do get MSTOKEN, and I can do "echo MSTOKEN" and it displays the Token, but I get this message.

"Net::IMAP::BadResponseError: Command received in Invalid state."

• Invoke redmine:email:receive_imap (first_time)
• Invoke environment (first_time)
• Execute environment
• Execute redmine:email:receive_imap
  rake aborted!
  Net::IMAP::BadResponseError: Command received in Invalid state.
  /var/lib/gems/2.7.0/gems/net-imap-0.2.3/lib/net/imap.rb:1247:in `get_tagged_response'
  /var/lib/gems/2.7.0/gems/net-imap-0.2.3/lib/net/imap.rb:1299:in `block in send_command'

mis org wrote:

What If I want to use different username, like helpdesks@ipc.org in different microsoft1.json file.

How can re-generate or re-initialize the (OAuth 2.0) token under different username in microsoft1.json (different .json file for different user)

getmail-gmail-xoauth-tokens --init microsoft1.json

This new username have proper access to redmine.

Please have a look at getmail's commandline options

--rcfile=FILE or -rFILE — read getmail rc file FILE instead of the default.

You can run multiple instances of getmail each with its own rc-file in which you specify individual retriever configurations.

mis org wrote:

When I try to initilize different .json file microsoft1.json with different username, I get this.

redmine@redmine-03:~/.getmail$ getmail-gmail-xoauth-tokens --init microsoft1.json
...
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 9 column 1 (char 597)

I don't know the content of your JSON file but the error seems to indicate that you forgot double quotes in line 9.

I've been following this thread, but how can I use getmail6 in Windows?

Lars M. wrote:

mis org wrote:

What If I want to use different username, like helpdesks@ipc.org in different microsoft1.json file.

How can re-generate or re-initialize the (OAuth 2.0) token under different username in microsoft1.json (different .json file for different user)

getmail-gmail-xoauth-tokens --init microsoft1.json

This new username have proper access to redmine.

Please have a look at getmail's commandline options

--rcfile=FILE or -rFILE — read getmail rc file FILE instead of the default.

You can run multiple instances of getmail each with its own rc-file in which you specify individual retriever configurations.

mis org wrote:

When I try to initilize different .json file microsoft1.json with different username, I get this.

redmine@redmine-03:~/.getmail$ getmail-gmail-xoauth-tokens --init microsoft1.json
...
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 9 column 1 (char 597)

I don't know the content of your JSON file but the error seems to indicate that you forgot double quotes in line 9.

Javier Martinez wrote:

I've been following this thread, but how can I use getmail6 in Windows?

See: Does getmail run on MS Windows?

Good luck with it.

Thanks! I'll check it up
Lars M. wrote:

Javier Martinez wrote:

I've been following this thread, but how can I use getmail6 in Windows?

See: Does getmail run on MS Windows?

Good luck with it.

Thanks to Lars M. for the clear guide.

I may just add one clarification for Office 365 :
- in your microsoft.json, the scope is defined as : "offline_access https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send"
- but on your Azure console, when you add API permissions (Microsoft Graph -> Delegated permissions), the namespaces do not match. example : https://graph.microsoft.com/IMAP.AccessAsUser.All

THIS IS NORMAL, don't try to change the scope in your microsoft.json, you will lose half a day on OAuth dark messages. (Dont' ask how I know...)

Lars M.

Thank you for the information.

1. When you get the token like 3 days back, it adds all three sections(access_token, expires_at, and refresh_token) in the new microsoft.json file, how do you when the token expires? I see this added in the microsoft.json file "expires_at": 1666890534.3506174,
How to auto renew the token ?

2. Once you run this command "getmail -r /home/redmine/.getmail/getmailrc", can you make it run all the time, even after a Ubuntu server reboot so it can auto process emails into Redmine.

getmail -r /home/redmine/.getmail/getmailrc & to run it in the background.

3. Does the email created has to be in certain format (Subject, Body) to be able to get process by getmail ?

I did sent a new email to helpdesks@ipc.org and then I run this command "getmail -r /home/redmine/.getmail/getmailrc", I get this message but I do see the new email showing up as issue in Redmine.

msg 16/17 (66528 bytes), delivery error (command rdm-mailhandler.rb 82444 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
Delivery error (command rdm-mailhandler.rb 82446 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
msg 17/17 (57668 bytes), delivery error (command rdm-mailhandler.rb 82446 error (77, Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.))
14 messages (1181061 bytes) retrieved, 3 skipped

Thanks again.

mis org wrote:

1. When you get the token like 3 days back, it adds all three sections(access_token, expires_at, and refresh_token) in the new microsoft.json file, how do you when the token expires? I see this added in the microsoft.json file "expires_at": 1666890534.3506174,
How to auto renew the token ?

This is something I still have to find out. I am not familiar with OAuth2. Maybe it is possible to configure and use a client_secret for this purpose? To best option is to ask this question in the getmail project.

2. Once you run this command "getmail -r /home/redmine/.getmail/getmailrc", can you make it run all the time, even after a Ubuntu server reboot so it can auto process emails into Redmine.

I explained a possible solution using Cron. But I did not test it yet, because I did not want to restart the VM. You could also start it as a system daemon.

3. Does the email created has to be in certain format (Subject, Body) to be able to get process by getmail ?

Yes, read the corresponding Redmine documentation: Forwarding emails from your email server. The errors are all from rdm-mailhandler.rb and have nothing to with getmail.

Nicolas Metters wrote:

Patrizio Evangelist wrote:

I Receive no TOKEN from the CURL command so what could be possibly wrong ?

[...]

The result of the command is : NULL ...

What should I put instead of DOMAIN too ? is it the Tenant ID ?

Once you declared your application in AzureAD:
- in the "Overview" you will have the application client ID and Tenant

- in "Add a Redirect URI", enable the Token V2 URL and set up your URI

- in certificates and secret, add a client
Beware: if you don't note down the secret when it's shown at creation, you won't be able to retrieve it later.
You'll be able to create oher additional clients but not recylcle the one you lost the secret for

Check out "How to connect to Office 365 with IMAP, Oauth2 and Client Credential Grant Flow"
It both a Youtube video and an article.
In the video, he show the Azure AD console and where he gets the information

Hi Nicolas,

First of all my Office 365 provider didn't authorise the access to this element : IMAP.AccessAsUser.All
=> This haver been fixed but the CURL script still not work :-(

Do you precisely know what element should be enabled on the AZURE configuration page to have IMAP working

A screenshot showing the list of all the required element should be fine, as well as the specific settings for each if there is some.
My Office 365 support are totally not familiar with REDMINE settings so they can't really help, and I can't change the settings myself.

Is there eventually a possibility to work on that problem in private by email, and when it will work drop a complete how to here so other users knows what they precisely have to do...

Thanks for your support.

Hi ,i also have this issue. my current bitnami version is bitnami 3.4.4.1. i'm using pop3 for receiving mail function .how can i fix this with termperarily solution ?many thanks.

Lars. M. Solution from Note 17. worked for me.

Using (OAuth 2.0) from IMAP to connect to office 365 mailbox using getmail6 on Linux Ubuntu 20.04 & Redmine 5.0.3.

https://www.redmine.org/issues/37688#note-17

Thanks,

-Arif

Lars M.

Are you sending emails to redmine via a gmail user mailbox OR an Office 365 user mailbox ?

Sending email from outlook 365 to user (Office 365 user mailbox) to create issue using getmail,

Is there special format we have to use in the subject and body section of the email ? the reason I ask, some email message auto creates issue in redmine, but some DO NOT.

Please advise?

Thank you again for your time,

-Arif

mis org wrote:

Are you sending emails to redmine via a gmail user mailbox OR an Office 365 user mailbox ?

Office 365 user mailbox

Sending email from outlook 365 to user (Office 365 user mailbox) to create issue using getmail,

Is there special format we have to use in the subject and body section of the email ? the reason I ask, some email message auto creates issue in redmine, but some DO NOT.

I think this is not an getmail issue. Please refer to the Redmine documentation on how to create issues by mail.

Please note, getmail will quit if an error occurs also if the error is harmless and can be recovered by a simple reconnect. Please read the proposed solution in Howto resume IDLE after error #147 to solve this.

Hey there!

I have the same issue than you and landed on your post trying to find a solution.

But before i can even do anything besides creating the files, i get an error.

which is that i can't even execute the first command "getmail-gmail-xoauth-tokens".
here is the error i get with a normal command block (in french) : ...\getmail6\configuration\data>getmail-gmail-xoauth-tokens ~/.getmail/provider.json 'getmail-gmail-xoauth-tokens' n’est pas reconnu en tant que commande interne ou externe, un programme exécutable ou un fichier de commandes.

I can't even go further, would you know where's the problem?

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

what is the probability that this fix comes in a 4.2.x version of redmine? Or do I have to take care of it myself?

I have done a commercial plugin for one of our customers, it works with 4.2 or 5.0 and is simple to use. Login in your configuration page with your Office 365 account.
If someone is interested to buy, contact me on info@cauliflyer.com.

I believe that anyone who has reactivated basic authentication within the Microsoft deadline will have no problems at first, based on: https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online.

Hi.

I found, tested and successfully deployed in production an easy workaround solution: Email OAuth 2.0 Proxy
Of course, one still needs to set up Azure App with appropriate Microsoft Graph permissions, configure proxy server and go through MS mail account activation process, but that's it. After deploying proxy server, Redmine was able to send and receive email through proxy via IMAP with basic authentication. No need for any Oauth2 stuff on Redmine side.

"Lars Method" works fine even with an older redmine: https://www.redmine.org/issues/37688#note-17. I used another server just to install getmail.

Andrew Beam wrote:

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

This is the best solution in my opinion. Works perfectly. Fast implementation. Easier and more reliable than getmail. Excellent work!

I've managed to configure sending and parsing received e-mails with OAuth2.0 proxy (https://github.com/simonrob/email-oauth2-proxy).

It took some time but it works as a service.

Who is interested in this proxy and needs help, feel free to contact me on mali.golub@gmail.com

I would be more than happy to have this in some future standard Redmine installations.

Julian Cookie wrote:

Andrew Beam wrote:

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

This is the best solution in my opinion. Works perfectly. Fast implementation. Easier and more reliable than getmail. Excellent work!

Hi,

Could you share how to set redirect URL,
Thanks!

Aria Kang wrote:

Julian Cookie wrote:

Andrew Beam wrote:

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

This is the best solution in my opinion. Works perfectly. Fast implementation. Easier and more reliable than getmail. Excellent work!

Hi,

Could you share how to set redirect URL,
Thanks!

I don't know what you mean. Do you refer to the URL you get, after you logged in to your MS365 Account after you copy/pasted the URL of the rake redmine:email:o365_oauth2_init into your browser?

Andrew Beam wrote in #note-62:

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

Hey,

I also installed the OAuth2 patch provided by Andrew Beam in #note-62 on Redmine 4.2.7. Thanks for this!

The installation went quickly in a few steps and works pretty well.
Only the installation guide is partially wrong and incomplete. So there were some hurdles, but succeed finally.

Many thanks.

Richard Hafenscher wrote in #note-72:

Andrew Beam wrote in #note-62:

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

Hey,

I also installed the OAuth2 patch provided by Andrew Beam in #note-62 on Redmine 4.2.7. Thanks for this!

The installation went quickly in a few steps and works pretty well.
Only the installation guide is partially wrong and incomplete. So there were some hurdles, but succeed finally.

Many thanks.

Hello, could you write instructions on how to install. I can't figure it out for the second day((...I will be very grateful
I stopped at the moment: token_file=/app/redmine/config/email_oauth2
I can not understand where to get this file or how to create it?

Hi,

I guess, you're talking about the init email account token rake command. I stuck at this point as well.
The token-file is created by this command, it doesn't exist before.

Consider to state a path, where you have write privileges.

After firing up this command, it stays exciting, because the documentation is missing, what to do exactly then.
The command gives you an URL back. Pick it and insert it into the browser. You get redirected. Copy the redirected address line now back into the console.

Then you can use the created token-file for authentication.
Note that the user who is running the receive email command needs write privileges on the token-file for recurrent updates.

If Anyone still interested i created Powershell App whitch can be deployed parall to container and use oficinall MS Graph API to Fetch email an pipe them to Redmine WS api.

https://github.com/steelants/Redmine-O365
Hope it can help someone :)

If you have any wuestuion just rise github issue :)

Andrew Beam wrote in #note-62:

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

It worked more succesfuly for me that method with getmail, but I stuck at the error:

Net::IMAP::BadResponseError: User is authenticated but not connected.

Already tried switch off IP6,
Checked/fixed this: https://stackoverflow.com/questions/74318775/unable-to-access-the-shared-mailbox-after-authenticated-user-is-authenticated
Yet no luck.

Anybody faced the same problem?

Andrew Beam wrote in #note-62:

https://github.com/beam/redmine-email-oauth wokrs without Getmail
PR welcomed

I'm trying to start this method. I can log in to mailbox but for each mail there is such error in log file:

MailHandler: an unexpected error occurred when receiving email: Access denied for user 'root'@'localhost' (using password: NO)

any ideas?

Janeks Kamerovskis wrote in #note-76:

It worked more succesfuly for me that method with getmail, but I stuck at the error:
Net::IMAP::BadResponseError: User is authenticated but not connected.

Anybody faced the same problem?

I am also stuck at "User is authenticated but not connected" error. Janeks Kamerovskis, have you been able to figure it out?

LE: I figured it out eventually. In my case, when validting the token code I was using a different O365 account (a domain admin) instead of the user actually configured in Redmine.