Slack to Mattermost on New Fedora server

Export everything from Slack

(Near end of free month : Rather urgent!)

Following these instructions : https://docs.mattermost.com/onboard/migrate-from-slack.html

Download the exported file (no permission required) and renamed to SLACK-EXPORT-FILE.zip = 12Mb

Create a Slack app in order to gather:

User emails
Uploaded attachment contents

OAUTH token for slack bot : export SLACK_TOKEN='lots-of-random-digits-and-letters-in-here'

https://github.com/grundleborg/slack-advanced-exporter/releases/

wget https://github.com/grundleborg/slack-advanced-exporter/releases/download/v0.4.0/slack-advanced-exporter.linux-amd64.tar.gz
tar -xzf slack-advanced-exporter.linux-amd64.tar.gz 

./slack-advanced-exporter --input-archive SLACK-EXPORT-FILE.zip \
                          --output-archive export-with-emails.zip \
                          fetch-emails --api-token ${SLACK_TOKEN}
# Was super quick (only 2 users + ?Google Drive)  = 12Mb+400Kb

./slack-advanced-exporter --input-archive export-with-emails.zip \
                          --output-archive export-with-emails-and-attachments.zip \
                          fetch-attachments --api-token ${SLACK_TOKEN}
# Lots of downloaded files = 1.120Gb now...

Note that this will only export your public channels. To capture your @dms (for instance), or if you aren't on a Pro account (though it's pretty slow), have a look at the "Fetching private DMs" section below.

We'll continue here with the MatterMost parts, so that there'll be evidence of movement towards something working. As noted below, the export/import process claims to have been done in such a way that you won't get duplicate messages, even if you have overlapping data.

Setting up the database

Install Postgres

https://docs.fedoraproject.org/en-US/quick-docs/postgresql/

dnf install postgresql-server zip # NOT NEEDED: postgresql-contrib

postgresql-setup --initdb --unit postgresql
#  * Initializing database in '/var/lib/pgsql/data'
#  * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log

systemctl start postgresql  # Start it now
systemctl enable postgresql # Set to run at boot

Set up Postgres for Mattermost

https://docs.mattermost.com/install/prepare-mattermost-database.html

sudo -u postgres psql
CREATE DATABASE mattermost WITH ENCODING 'UTF8' LC_COLLATE='en_US.UTF-8' LC_CTYPE='en_US.UTF-8' TEMPLATE=template0;
# Replace 'mmuser-password' with something more secure
CREATE USER mmuser WITH PASSWORD 'mmuser-password';
GRANT ALL PRIVILEGES ON DATABASE mattermost to mmuser;
ALTER DATABASE mattermost OWNER TO mmuser;
GRANT USAGE, CREATE ON SCHEMA PUBLIC TO mmuser;
\q

systemctl restart postgresql  # 'adsorb' the changes?

Add 'Trust' : Local Database (same server)

If the Mattermost server and the database are on the same machine:

Open /var/lib/pgsql/data/pg_hba.conf as root in a text editor

Find the following lines:

local   all             all                        peer
host    all             all         ::1/128        scram-sha-256

Change peer and ident to trust:

local   all             all                        trust
host    all             all         ::1/128        trust

systemctl reload postgresql

Verify that `mmuser` works

psql --dbname=mattermost --username=mmuser --password
# Seems to work even if password is incorrect...  Need a better test
\l # is 'show databases' in MySQL
\c mattermost
# Something else?
\q  # quit

Install Mattermost itself

https://docs.mattermost.com/install/install-rhel-8.html

As the fedora user (arbitrary):

wget https://releases.mattermost.com/10.0.0/mattermost-10.0.0-linux-amd64.tar.gz
# 580Mb download
tar -xvzf mattermost*.gz
sudo mv mattermost /home  # So /opt/mattermost is /home/mattermost
sudo mkdir /home/mattermost/data
sudo chown -R mattermost:mattermost /home/mattermost
sudo chmod -R g+w /home/mattermost
sudo useradd --system --user-group mattermost --home-dir /home/mattermost
sudo touch /lib/systemd/system/mattermost.service

As root, add the following to /lib/systemd/system/mattermost.service :

[Unit]
Description=Mattermost
After=postgresql.service
BindsTo=postgresql.service

[Service]
Type=notify
ExecStart=/home/mattermost/bin/mattermost
TimeoutStartSec=3600
KillMode=mixed
Restart=always
RestartSec=10
WorkingDirectory=/home/mattermost
User=mattermost
Group=mattermost
LimitNOFILE=49152

[Install]
WantedBy=multi-user.target

Edit the configuration file

cp /home/mattermost/config/config.json /home/mattermost/config/config.defaults.json
sudo -u mattermost joe /home/mattermost/config/config.json

Configure the following properties in this file:

Set DriverName to "postgres". (This was the default)
Set DataSource to "postgres://mmuser:<mmuser-password>@<host-name-or-IP>:5432/mattermost?sslmode=disable&connect_timeout=10" replacing mmuser, <mmuser-password>, <host-name-or-IP>, and mattermost with your database name.
Set your "SiteURL": The domain name for the Mattermost application (e.g. https://mattermost.example.com).

  \\... ~first line
  "SiteURL": "https://mattermost.example.com",
  \\... further down
  "DriverName": "postgres",
  "DataSource": "postgres://mmuser:mmuser-password@localhost:5432/mattermost?sslmode=disable&connect_timeout=10",

Start up Mattermost

systemctl start mattermost
systemctl enable mattermost.service

To confirm Mattermost is running, as any user:

curl http://localhost:8065
# Should see HTML output

`certbot` for Let's Encrypt Certificate(s)

https://reintech.io/blog/securing-apache-lets-encrypt-fedora-38

dnf install certbot python3-certbot-nginx

#certbot --nginx -d mattermost.example.com
# Cannot work, since nginx needs the certificate file in its config

certbot -d mattermost.example.com certonly
# Can work with option '2' = temporary webserver for authentication
# Note down certificate locations:
#Successfully received certificate.

#Certificate is saved at: /etc/letsencrypt/live/mattermost.example.com/fullchain.pem
#Key is saved at:         /etc/letsencrypt/live/mattermost.example.com/privkey.pem

These certificate filenames should be copied into lines ~20-22 of the nginx configuration below.

Set certbot to renew automatically using systemd timers:

systemctl status certbot-renew.timer
# Yes - it exists...
systemctl enable --now certbot-renew.timer

Actual timer specification in /etc/systemd/system/timers.target.wants/certbot-renew.timer

`nginx` configuration

dnf install nginx

https://stackoverflow.com/questions/78195706/nginx-v1-25-1-http2-on-new-recommendation-creating-another-warn

Add to /etc/nginx/conf.d/mattermost.conf (adjust example.com references to your own server) :

upstream backend {
   #server 10.10.10.2:8065;
   server 127.0.0.1:8065;
   keepalive 32;
}

server {
  listen 80 default_server;
  server_name   mattermost.example.com;
  return 301 https://$server_name$request_uri;
}

server {
   server_name    mattermost.example.com;

   listen 443 ssl;
   listen [::]:443 ssl;
   http2 on;

   http2_push_preload on; # Enable HTTP/2 Server Push

   # Adjust the following two lines for your mattermost domain / path to key files
   ssl_certificate /etc/letsencrypt/live/mattermost.example.com/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/mattermost.example.com/privkey.pem;
   ssl_session_timeout 1d;

   # Enable TLS versions (TLSv1.3 is required upcoming HTTP/3 QUIC).
   ssl_protocols TLSv1.2 TLSv1.3;

   # Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
   # prevent replay attacks.
   #
   # @see: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
   ssl_early_data on;

   ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:50m;
   # HSTS (ngx_http_headers_module is required) (15768000 seconds = six months)
   add_header Strict-Transport-Security max-age=15768000;
   # OCSP Stapling ---
   # fetch OCSP records from URL in ssl_certificate and cache them
   ssl_stapling on;
   ssl_stapling_verify on;

   add_header X-Early-Data $tls1_3_early_data;

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60s;
       send_timeout 300s;
       lingering_timeout 5s;
       proxy_connect_timeout 90s;
       proxy_send_timeout 300s;
       proxy_read_timeout 90s;
       proxy_http_version 1.1;
       proxy_pass http://backend;
   }

   location / {
       client_max_body_size 2000M;  # This is super-large for slack import : Reduce later
       proxy_set_header Connection "";
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_http_version 1.1;
       proxy_pass http://backend;
   }
}

# This block is useful for debugging TLS v1.3. Please feel free to remove this
# and use the `$ssl_early_data` variable exposed by NGINX directly should you
# wish to do so.
map $ssl_early_data $tls1_3_early_data {
  "~." $ssl_early_data;
  default "";
}

systemctl enable --now nginx.service

Mattermost configuration

Some ideas : https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2236

Once an initial user is created on the web interface:

Go to : "Mattermost" (at the top) - "System Console"
- "Authentication" - "Signup"
  - Enable Account Creation = False
  - SAVE
- "Authentication" - "Email"
  - Require Email Verification = False # Needed for mmctl below
  - SAVE
- "Site Configuration" - "Customization"
  - Site Name = 'XYZ'
  - Enable Custom Branding = True
  - Custom Brand Image = (upload)
  - SAVE
- "Environment" - "SMTP"
  - Thankfully there's a 'test configuration' button
  - Iterate on the settings until it works (may need STARTTLS, for instance)

Import slack data into Mattermost

https://docs.mattermost.com/onboard/migrate-from-slack.html part 4

Download the import conversion tool from https://github.com/mattermost/mmetl/releases/ :

wget https://github.com/mattermost/mmetl/releases/download/v0.1.1/linux_amd64.tar.gz
mv linux_amd64.tar.gz mmetl_linux_amd64.tar.gz
tar -xzf mmetl_linux_amd64.tar.gz

Make sure that your admin user is different from the username coming in from Slack : It could cause hassle if there's an overlap!

./mmetl transform slack --team aiadventures \
                        --file export-with-emails-and-attachments.zip \
                        --output mattermost_import.jsonl
# Finishes with 'transformation successful!'

zip -r mattermost-bulk-import.zip data mattermost_import.jsonl
# Output file is 'mattermost-bulk-import.zip'

Good to know : "The migration is idempotent, meaning that you can run multiple imports that contain the same posts, and there won’t be duplicated created posts in Mattermost. "

Check that the file size allowed by Mattermost exceeds the size of the zip :

https://forum.mattermost.com/t/solved-mmcil-cloud-to-self-host-migration-error-failed-to-create-upload-session-unable-to-upload-file-file-is-too-large/16372
Go to : "Mattermost" (at the top) - "System Console"
- "Environment" - "File Storage"
  - Maximum File Size = 2000 (i.e. 2Gb is larger than my zip)
  - SAVE
/home/mattermost/bin

# Check that we have the tool
sudo -u mattermost /home/mattermost/bin/mmctl --help
# Configure authentication:
#   https://docs.mattermost.com/manage/mmctl-command-line-tool.html#mmctl-auth
sudo -u mattermost /home/mattermost/bin/mmctl auth login --help
sudo -u mattermost /home/mattermost/bin/mmctl auth login https://mattermost.example.com --name admin --local
# Need admin account (first account) username / password
# 'credentials for "admin": "user-admin@https://mattermost.example.com" stored'

sudo -u mattermost /home/mattermost/bin/mmctl import upload ./mattermost-bulk-import.zip
# Upload session successfully created, ID: 9waazdc1tb8k3exuzgc9rcc1hi 
# Import file successfully uploaded, name: k5ptf7nmqibh5xigbi4z7atsym

sudo -u mattermost /home/mattermost/bin/mmctl import list available
# 9waazdc1tb8k3exuzgc9rcc1hi_mattermost-bulk-import.zip
# Shows the import file exists
#   It's actually just in /home/mattermost/data/import/
#     ... could we have just put it there?

sudo -u mattermost /home/mattermost/bin/mmctl import process 9waazdc1tb8k3exuzgc9rcc1hi_mattermost-bulk-import.zip
# Import process job successfully created, ID: rqd8mrzd1jratcysjcc4my6m7o

# And off it goes!

mmctl import job show --json rqd8mrzd1jratcysjcc4my6m7o
# After a while:
#     "status": "in_progress",
# becomes ...
#     "status": "success",

Use the imported team

During the import process, the emails and usernames from Slack are used to create new Mattermost accounts.

Slack users can activate their new Mattermost accounts by using Mattermost’s Password Reset screen with their email addresses from Slack to set new passwords for their Mattermost accounts.

Once logged in, Mattermost users will have access to previous Slack messages in the public channels imported from Slack.

ALL DONE!

Fetching private DMs

https://github.com/rusq/slackdump/blob/master/doc/compiling.rst

git clone github.com/rusq/slackdump
cd slackdump
go build ./cmd/slackdump
# Downloads and builds

./slackdump --help
# Will give nice page of text if it works

https://github.com/rusq/slackdump/blob/master/doc/login-manual.rst

Make a `secrets.txt`

Add this to secrets.txt, alongside the ./slackdump executable:

SLACK_TOKEN=xoxc-
COOKIE=xoxd-

This is a special filename, that will get searched for...

Get the `SLACK_TOKEN`

In the javascript developer console of a logged-in Slack session:

JSON.parse(localStorage.localConfig_v2).teams[document.location.pathname.match(/^\/client\/([A-Z0-9]+)/)[1]].token

Save the token (starts with xoxc-) somewhere.
Copy/paste that into secrets.txt

Get the `COOKIE` value

Switch to Application tab and select Cookies in the left navigation pane.
Find the cookie with the name "d". That's right, just the letter "d".
Copy/paste the 'URL decoded value' into secrets.txt

Get the channels/DM IDs of interest

Now the ./slackdump will use the secrets.txt file implicitly.

https://github.com/rusq/slackdump/blob/master/doc/usage-list.rst

Get the channels of interest ... :

# ./slackdump -list-channels # Old-style (obsolete)
./slackdump list channels  # New v2 format
# We need just the IDs (not the @dm annotations) - select the channels you need
# eg:
C565QKKTQ
D3S85G2CE
D3QAXQ2SD

Put this list in dms.list - should be a newline separated list of the first column of IDs.

Now download the conversations and attachements

https://github.com/rusq/slackdump/blob/master/doc/usage-export.rst#id4

./slackdump export -o slackdump_export_dms.zip -type mattermost @dms.list

Transform them into a Mattermost format

On the server, now, create a new directory dm_import (for instance), and go into it. Put slackdump_export_dms.zip here.

<PATH.TO>/mmetl transform slack --team aiadventures \
                         --file slackdump_export_dms.zip \
                         --output mattermost_import.jsonl

# INFO[0003] Exporting posts                              
# INFO[0003] Transformation succeeded!

This will generate a directory bulk-export-attachments and mattermost_import.jsonl file in the current directory.

Create a zip archive in bulk format:

zip -r bulk_import_dms.zip data mattermost_import.jsonl

./mmctl import upload ./bulk_import_dms.zip
./mmctl import list available | grep bulk_import_dms.zip
# Will output a strange ID ending in zip
./mmctl import process THAT_ZIP_ID
# Will output another strange ID
./mmctl import job show THAT_ID

@DMs are now imported *